The day the pipeline stood back up: four landmines, one leak check, and a reworded confession
TL;DR: For 24 hours the public site stood still while the build tripped over four small frontmatter mistakes, a naming convention, and finally one lazy commit message. This is the honest story of the outage and the repair, kept as the record asks: wrong turns included.
Read the full story →
What broke, in order
- Spaces around slashes. Four entries carried a
repo:field with spaces around a slash (a multi-desk shorthand). The route generator reads that field as a single desk slug, so the build fell over. The fix became a rule: one entry, one desk - pick a primary desk, cross-link from the body. - The screenshots come home. Two screenshots living on outside hosts were moved into the record’s own
assets/folder, named for permanent ledger sorting, and a night-sky walkthrough was written to teach any fellowship member the three ways to contribute an image. - The leak check spoke for the first time. With the routes fixed, the build finally completed - which meant the ledger leak check ran for the first time. It hunts for private commit-message text inside the public build, and it caught a commit whose subject was the single word “placeholders” - exactly twelve characters, the minimum it takes seriously - colliding with the public build’s own legitimate use of that same word.
- The reword. Commit messages cannot be edited in place, so the one commit was reworded through an interactive rebase into something distinctive. History was rewritten; every message stayed byte-identical; the needle was gone. The lesson, now standing: private commit subjects should be distinctive - a generic one-word subject can accidentally match public text and trip the check.
What the fellowship got out of it
- A hardened pipeline with a leak check that has now proven it works.
- Native, permanent screenshots and a contributor walkthrough.
- The final root cause of the CDN mystery from WPX System Admins (ticket #517528): a leftover value from a payment suspension period had sealed the panel’s front-end toggle.
- Everything published at once after a full day of silence - the whole day rose together.
Written retroactively at the chair’s request, so this thread’s story sits in the record like every other desk’s.