The ground has shifted. Bots have become a major scourge. To keep sites fast, reliable, and secure, we’re focusing resources on adapting to these new demands. And we need your help.
Automated crawlers and bot traffic have reached unprecedented levels. Across many Newspack sites, bots now generate over 90% of all incoming requests. This surge in bot activity is an industry-wide problem affecting publishers across all platforms.
The issue is not well-known crawlers that identify themselves or respect robots.txt files, or even AI scrapers that typically hit our “edge cache” where the volume is easy to manage. The most concerning bots are those that disguise themselves as legitimate traffic but whose purpose is unclear. For a sense of the scale, one site recently logged 7,400 pageviews from readers but a total of 2.2 million requests to our infrastructure in a single day.
Newspack’s engineering team, together with Automattic’s systems team – which manages our cloud service – handles these bots every day and works behind the scenes to throttle and block them. That’s part of the service you pay for.
But the surge in the number of requests has exposed a secondary vulnerability: namely, unapproved plugins with potential performance issues that only show up under extreme load.
Historically, Newspack has had a flexible approach to third-party plugin installation. But increasingly, even a minor unsupported setting or uncommon plugin can significantly degrade site performance if bots are hammering the site.
That presents fresh challenges. And some of you may have experienced recent slowness in your sites as a result.
But our goal remains the same: to ensure that all Newspack sites always operate at peak readiness, because nobody knows when a breaking news event could happen, particularly headed into election season.
What we’re doing
While we will continue to respond to individual reports of site slowness, the problem demands a systematic approach.
The team is focusing on hardening stability, security, and the tooling we use to keep your site healthy, so that sites are capable of holding up in the modern internet ecosystem.
Specifically, we are:
- Working with the systems team to optimize and better prepare your site and our cloud service for the current conditions, including provisioning extra capacity as needed.
- Reviewing and auditing third-party plugin installs to identify unnecessary or redundant functionality. There are 1,521 active plugins on live Newspack sites, and about 1,000 of which only run on a single site. Many duplicate something Newspack already does well, like popup builders, stats plugins, and form builders.
- Turning off settings we know don’t scale. A handful of plugins we recommend have configuration options that can cause problems under load. We’re disabling and hiding those options.
- Tightening third-party plugin installation. We’ll bring the “Allowed Plugins” list up to date, improve the plugin review process, and harden some of the third-party plugin install flows.
- Improving internal monitoring, alerting, and bot blocking tools.
What this means for you
- You may notice improvements in site response times over the next few weeks and fewer periods where the site feels sluggish.
- We expect limited impact on our regular release cadence. Migrations and in-flight project work continue as planned, but lower priority features may take a little longer.
- You may hear from us about third-party plugins you’ve installed. We’ll explain what we found and, where a Newspack feature already covers the same need, we’ll show you how to use it. This would be a great time to audit your plugins and remove any that you’re not using, both active and inactive.
- If you’re considering installing something new, consider the implications first. Every new plugin has an impact on performance, including the vibe-coded ones! Be sure to talk to a TAM or submit it through the plugin review form first.
If your site feels unbearably slow, is timing out, or throwing errors, please tell us immediately via the Help Channel in Newspack Slack. Those reports are the most useful thing you can give us over the next two weeks, and they feed straight into this work.
We’ll follow up soon with what’s changed and what we found.