The great slowness investigation

The symptoms

On a cold cache, pages took 3-11+ seconds to generate (curl-measured TTFB: /contact/ 11.4s, /resources/ 9.3s). Cached pages were fast (~150-400ms). Worse: ~30 parallel requests to warm the cache after a plugin change and more than half timed out at 10 seconds with zero bytes - not slow, simply absent.

The ceilings

Ilia R. (Technical Support Shift Lead) confirmed the plan’s shape: 3 PHP workers per site, and a 200 requests/minute rate limit that lives in the DDoS layer for the whole server and can’t be raised per-account. Modest numbers - until something multiplies work.

The first multiplier: wp-cron

The logs showed wp-cron events firing more than once per second. WPX disabled automatic wp-cron and set a real server-level cron every 5 minutes - though, as Christopher later caught, that cron initially covered only the service subdomain; the main site’s queue sat with 20+ overdue scheduled events that never ran even when wp-cron.php answered with a clean 200. That gap became its own follow-up.

The second multiplier: bots on the calendar

Georgi Y. (Security & Optimization) traced the CPU spikes to automated crawlers hammering The Events Calendar’s dynamic query strings (eventDisplay, posts_per_page, tribe-bar-date) - URLs that bypass page caching and force real PHP work per hit. WPX deployed a targeted CAPTCHA challenge on those query strings, and CPU stabilized. The calendar plugin was deactivated site-wide as a stopgap.

The root cause

Then the real mechanism surfaced: W3 Total Cache’s “Cache URIs with query string variables” setting (pgcache.cache.query) was false in the saved config on both sites - so any URL with a query string skipped the page cache entirely and was rebuilt from scratch on every request, calendar or not. The calendar wasn’t special; it just generated more query-string URLs than anything else. The setting was greyed out in W3TC’s own UI (a suspected side effect of Varnish being enabled at 127.0.0.1:6081), which is what had kept it invisible. The resolution - and the calendar’s return - is told in the publicrelations site entry.

The mystery solved in passing

Mid-investigation, a separate ghost: a WPCode footer snippet on the service site, confirmed correct in the database, kept serving its old version byte-for-byte even on responses marked x-cache-status: MISS. The Varnish hypothesis fell (WPX confirmed the config; the CDN wasn’t the culprit; the dashboard cache-clear test ruled it out). The real culprit was WPCode’s own internal snippet cache, which never refreshed on edit. Forced rebuild, instant fix. A fine reminder: sometimes the stale layer is the plugin you forgot has a cache of its own.

Also logged while the door was open: publications carried two A records (16 and 34 - harmless, but noted), and the brand-new publicrelations subdomain had only a self-signed placeholder cert while the real one was pending.