Hey all — after doing some upgrades on the site backend, I’m starting to once again see the old problem crop up with the SCW homepage or the Eyewall homepage not updating when @Eric or @Matt put up new posts. (Both sites share hosting, so any configuration issues tend to affect both sites.) The problem is annoyingly transitory and doesn’t happen all (or even most) of the time, but I have caught it once or twice in the past month.
If you do see the problem behavior happening—that is, if you hit the SCW or Eyewall homepages and don’t see the day’s latest post at the very top—please reply below and let me know when. This’ll help me know where to look in the server logs to see if I can spy out any issues!
Tap to expand for a whole bunch of geeky talk on what might be going on and how you can help.
What might be going on
As usual, this is a caching issue. (It’s always cache, except when it’s DNS, ugh.) For a quick refresher, we have two layers of cache that are relevant here:
-
The on-box nginx fastcgi cache. Nginx is the application that actually dishes up Wordpress-generated pages. Nginx runs on our web server, which is itself an AWS virtual machine in AWS’ Ohio datacenter.
-
Cloudflare’s cache, which lives on Cloudflare’s network.
When Eric or Matt hit “publish post” in Wordpress, a couple of automated actions are triggered to ensure that the new post appears immediately on the homepage:
-
First, wordpress sends a command to nginx to remove (“invalidate,” in cache lingo) the current homepage copy it’s holding in its local on-box cache;
-
Then, wordpress sends a similar command to cloudflare telling it to invalidate its current copy of the site homepage that cloudflare is holding onto in its big global distributed edge cache.
When a visitor like you next loads the homepage, your request first goes to Cloudflare. Cloudflare sees that it has no cached copy of the homepage, so it pokes the web server and asks for one. The web server sees that it too has no cached copy of the homepage, so it asks Wordpress to cook up a fresh one. That fresh homepage is cached by nginx, sent to cloudflare, cached again there, and finally served to you.
I think the issue we’re running into might be a classic race condition: it seems like sometimes—not all the time, but just often enough to be annoying—nginx’s locally cached version of the homepage is being invalidated after cloudflare’s is invalidated. This leads to a situation where cloudflare has no cached version and requests one from nginx, which, still having the old version in cache, serves that back to cloudflare. This causes the old version of the homepage to be cached at the edge by cloudflare and served out to visitors until it expires from inactivity, or Matt or Eric or Dwight or I actively clear it from Cloudflare’s cache and force Cloudflare to fetch a new version again.
I don’t have much of a fix in mind yet. If that is indeed what’s happening, I suspect what I’ll need to do is create an mu-plugin for wordpress that on story publication performs an additional forced local nginx cache invalidation call and a subsequent cloudflare edge cache invalidation call several seconds after the post goes up. This ought to clobber whatever race condition might be happening with another invalidation of both caches in known-good order. That’s my hope, anyway.
How you can help
If you hit the SCW or Eyewall homepages and see a stale version of the homepage missing the latest post, open up your browser’s developer console and look a little deeper. (It’s the F12 key for Chrome, Brave, and Firefox; for Safari, go option-cmd-I.) Select the “network” tab and reload the homepage. You’ll see a list of all the different network calls your browser had to make to display the page. Scroll to the very top and look for the initial page load request:
Click that request to open up an inspector window to the right. We want to look at the “response headers”—that is, the little bits of metadata the web server attaches to each request to tell your browser what it’s doing. Specifically, we’re looking for a header named CF-Cache-Status. You should see “HIT” there, indicating the page was served from Cloudflare’s cache. You also might occasionally “MISS” if Cloudflare has just invalidated its cache and had to reach back to the web server. Either of those is OK:
Scrolling down a bit, you’ll see a second header called X-Nginx-FastCGI-Cache. This one will also either say “HIT” if nginx pulled the page from cache, or “MISS” if it had to get the page fresh from Wordpress:
You can also tick the “Disable Cache” selection box to force your own web browser to demand fresh pages from Cloudflare. Doing this and refreshing will then cause Cloudflare’s cache status to show as “BYPASS,” meaning Cloudflare ignored its own cache and served up exactly what our nginx server had for it, cached or no:
From these three things, we can tell a lot about what’s stale where.
Case 1: You select “Disable Cache” and refresh and suddenly you see a new shiny correct version of the homepage.
If this happens, the issue is that Cloudflare seems to have failed to invalidate its cache properly. You can try clearing your browser’s cache, but the best thing to do is to post in this thread and let me know so I can manually dump cloudflare’s cache. It’ll also eventually expire on its own and update itself, but it might take several minutes (or even hours.)
Case 2: You select “Disable Cache” and refresh but you’re still seeing a stale version of the homepage.
If you have “Disable Cache” selected and you’re refreshing but still seeing a stale homepage, look at the Nginx cache response header. If it says “HIT,” then that’s probably the race condition problem behavior manifesting: Nginx has a stale page in its cache that it shouldn’t, and Cloudflare is dutifully caching and serving out whatever nginx gives it. Definitely post in the thread here and let me know.
Case 3: You’ve got “Disable Cache” selected, Cloudflare’s header reports “bypass,” nginx’s reports “miss,” and you’re still seeing a stale homepage
This one might be a local browser cache issue, or it might be some really weird craziness going on. If clearing your local browser cache doesn’t fix it, then definitely post here and let me know!
Thanks for reading all of that, and thanks for being patient. Hopefully we can nail this silliness down once and for all!