Short version. The 10:30 reports were wrong because the code that sends them was never the code that had been fixed. Three separate times in four days, a release was pinned that was cut from a side branch instead of main. Two of those pins killed the sender outright and nothing told anyone. When I opened this at 2 PM today, the sender was dead again and would have skipped tomorrow's Houston, BCS, and Dallas reports. That is repaired and proven through the real scheduler. The plan below removes the hand-operated release step that caused all of it.
| Symptom | Real cause (verified) | State now |
|---|---|---|
| Houston "0 of 3 reported" while HOURP, HOUZN, HOUUS all had good rows | The release sending at 10:30 (b79e6a56) was cut at 10:09 from a Sept 1 side branch. It never contained the Aug 26 Houston fix (commit 09cdc116, which still sits on a Codex branch and was never merged). Two unrelated failed intake files (a HOUZN departures CSV and a HOUUS A/R aging PDF) were treated as a market-wide failure and wiped all three hotels. [S1][S10] | Scope fix is on main (commits 0ff9c507, 621c95d0) and in the live release as of 14:12 CT. Houston dry run renders "All 3 reported". Fixed, awaiting tomorrow's real send. [S10][S4] |
| BCS "5 of 7", CLLDI (Howdy) missing | The Howdy Flash PDF prints a small arrow between TODAY and MONTH TO DATE; the parser rejected it. The direct Cloudbeds occupancy link was checked once, before the export was ready, and abandoned. Woz tried once at 05:17, changed nothing, then was stuck on that same item for every 5-minute wake until cutoff. [S1][S9] | Parser accepts the arrow, link download now polls for readiness (f74411c8), Woz advances past attempted items (d1f6b5f0, 29 unit tests pass). CLLDI Sept 3 row exists: 98 rooms, 26.53% occupancy, $51.41 ADR. Fixed, awaiting tomorrow. [S16][S10] |
| "All the CLLDI reports were scheduled and still not delivering" | The handoff note said only 2 of 5 Cloudbeds reports arrived. That is wrong. All five arrived from Cloudbeds between 03:01 and 03:02 CT. Four of the five were stored with no property code (Adjustments/Voids, Payment Processing, Daily Revenue, one Daily Financial), so they were invisible to anything looking for CLLDI, and none of the four has a parser. Cloudbeds did its job; our intake dropped the label. [S6] | Not yet fixed. In the plan (item 3.3). |
| "Where was Woz's check?" | Two answers. (a) The Woz worker ran every 5 minutes but its queue selector re-picked the one item it had already tried, so it never looked at anything else. (b) Bigger: Market Pulse delivery failures are routed to "Woz" in the alert system, but the only unattended Woz worker is hard-limited to report-intake issues. 16 Market Pulse alerts have been sitting there since Aug 23 with nothing able to act on them. By the Aug 7 design, none of them reach a person. [S9][S8][S14] | (a) fixed in d1f6b5f0. (b) structural gap, in the plan (item 2.3). |
At 14:03 CT today, all three Market Pulse jobs and the freshness monitor were failing before they reached the sender. The release pinned after the corrections (67524634, correctly cut from main) failed the egress safety gate, because main never received the two "retire the Woz Discord route" commits that only existed on this morning's side branch, and a new unloaded PIP digest script on main contained the literal channel-variable name the gate scans for. The gate fails closed, the launcher stops, and nothing sends. Tomorrow's Houston, BCS, and Dallas reports would not have gone out. [S3][S4][S5]
This is the exact mechanism that killed Sept 1 to 3: a pin from a branch that lacked the retirement commits. Codex's own restore note predicted it would recur if the commits were not merged to main. They were not merged. [S2]
| When (CT) | Pointer moved to | Cut from | Result |
|---|---|---|---|
| Sept 1, 19:08 | cf96969c | codex/clldi-cloudbeds-dash side branch | Egress gate fails. Sender dead Sept 1, 2, 3. No alert reached a person. [S2] |
| Sept 4, 10:09 | b79e6a56 | cf96969c plus 3 commits (still off main, 59 main commits behind) | Gate passes, sender runs, but ships the unfixed Houston scope bug at 10:30. [S10][S7] |
| Sept 4, 11:28 | 67524634 | main | Pointer written as a symlink, not a text file. Woz worker and the egress check crash on every run until 13:46. [S9][S4] |
| Sept 4, 13:46 | 67524634 | main (pointer repaired to a text file) | Egress gate fails on main's own contents. Sender dead again. Reported as healthy. [S5][S3] |
| Sept 4, 14:12 | d337bced | main, after merging the retirement commits and removing the literal token | All four launcher gates pass, egress PASS, Houston dry run "All 3 reported", all three jobs exit 0 through launchd, Woz worker reads the new release. [S5][S3][S11][S12] |
| Change | Proof |
|---|---|
| Merged the two egress-retirement commits into main and rewrote the PIP digest launcher so no literal Woz channel name remains. Commit d337bced. | Egress check against the tree: PASS, zero failures. Launch-contract, failsafe, and verification tests: 27 pass, 0 fail. GitHub deploy run 33909664974: success. [S5][S10][S11] |
| Cut immutable release revparpro-market-pulse-d337bced from main and repointed the runtime. Previous pointer value saved. | Identity gate ok, hot-patch gate clean, roster gate 14, egress PASS. Pointer is a regular text file. [S12][S5] |
| Proved the real scheduled path, not a dry run of my own. | launchctl kickstart of houston, bcs, dallas jobs: all exit 0, each logged "already sent for 2026-09-03" and sent nothing new (dispatch table confirms zero new sends). Freshness monitor passed its egress gate and ran its controls. Woz worker receipt at 19:14Z names the new release. [S3][S4][S7][S9] |
| Corrected Houston and BCS reports already in the original Gmail threads (done by Codex before this review, verified by dispatch records). | Houston 10:30 dispatch 8be0a446 and BCS 044676bc exist; corrections were sent into the same threads per the incident record. [S7][S1] |
Not proven yet, and I will not call it fixed until it is: tomorrow's real 03:00 Cloudbeds cycle, the Woz morning window (05:00 to 10:34 CT) advancing past attempted items, and the 10:30 Houston and BCS sends showing the right hotel sets in the actual inboxes. I will be watching each step live tomorrow morning.
Owner is Woz for everything below unless marked as your decision. "Proof" is what has to be true before I say done.
| # | Action | Proof of done |
|---|---|---|
| 1.1 | Live repair of the dead sender (done, section 3). | Done and proven through launchd. |
| 1.2 | Watch tomorrow's cycle end to end: 03:00 Cloudbeds five-report set, Woz claims and advancement 05:00 to 10:34, readiness window, 10:30 sends. | Houston shows 3 of 3, BCS shows CLLDI, in the recipients' Gmail threads, with no operator touch. This is the literal closure test of the open incident. |
| 1.3 | Built and loaded today (14:26 CT). Unattended release-contract preflight runs at 08:00 and 10:00 CT: pointer is a text file, release name matches its commit, no hot-patch drift, commit is on origin/main, egress PASS, Woz worker points at an existing release, all three market jobs point at the launcher and fire at 10:30. It writes a receipt and exits red on any failure. [S18] | Green on the live release (11 checks, 0 failures). Red on all three shapes that bit us: an off-main pin (b79e6a56), a symlink pointer, and today's broken main pin (67524634). Tomorrow's 08:00 and 10:00 receipts must be green. Auto-rollback and alerting come with item 2.2. [S18] |
| # | Action | Proof of done |
|---|---|---|
| 2.1 | Promote script. One command takes a commit, refuses unless it is on origin/main, cuts the release folder, runs every gate (identity, hot-patch, roster, egress, launch-contract tests, and a dry-run render for each market), writes the pointer atomically as a text file with a backup, kicks the three jobs plus the Woz worker, and prints a receipt. Any other way of moving the pointer is a policy violation and the detector below catches it. | A deliberate attempt to promote an off-main commit is refused. A test promotion of main passes and the receipt shows all gates green. |
| 2.2 | Release-contract detector, every 15 minutes, in its own job, not inside the freshness monitor and not behind the egress gate: pointer type, on-main ancestry, egress PASS, Woz plist target exists, last exit codes of the three market jobs and the Woz worker. On failure it rolls back to the last known-green release automatically and records the event. | Break the pointer on purpose (symlink, off-main path); within 15 minutes it is rolled back and logged. |
| 2.3 | Give Market Pulse alerts an actor. Either the Woz worker gains a bounded recipe for the two Market Pulse routes (re-run preflight, roll back release, retry send within the window) or those routes stop being assigned to a worker that cannot take them. Drain or close the 16 stale rows. | Zero Market Pulse alert rows older than one business day with no action recorded. |
| 2.4 | Dead-sender notice (your decision, see section 7). If no Market Pulse dispatch exists by 10:45 CT, a separate tiny job (not the sender, not the freshness monitor) sends the recipients a one-line "Market Pulse delayed, being repaired" note instead of silence. | Simulated missed send produces the note; a normal day produces nothing. |
| 2.5 | Remove the stale Aug 1 fallback folder the launcher silently uses when the pointer is unreadable. Prune release folders to the last three and clean the 216 worktrees. | Unreadable pointer now refuses to send instead of running month-old code. Release count is 3. |
| # | Action | Proof of done |
|---|---|---|
| 3.1 | Houston and BCS scope fix (unrelated intake failures can no longer wipe a market; a hotel-scoped failure affects only that hotel). | Tomorrow's real 10:30 sends, plus the existing unit tests. |
| 3.2 | Woz evidence packet gains a bounded, credential-free excerpt of the source text so a parser-layout defect (like the arrow) is fixable in one attempt instead of a blind retry. | Replay the Sept 4 CLLDI item through the worker: it produces a repair candidate, not "retry". |
| 3.3 | Cloudbeds attribution and completeness. Every message from the Howdy subscription is stamped CLLDI regardless of subject. A per-morning check expects exactly five direct reports and names which ones are missing. Daily Financial and Daily Revenue become parseable fallbacks so CLLDI's night audit no longer hinges on one link download plus a front-desk email. | Sept 5 morning: five rows, all CLLDI, all in a terminal state. A simulated missing subscription is named in the check output. |
| 3.4 | Fix misleading sender log lines: a re-check on an already-sent day currently logs "sending truthful degraded report" while sending nothing. That wording nearly caused me to suspect a duplicate send today. | Log says "already sent, re-check only" on that path. |
| # | Action | Proof of done |
|---|---|---|
| 4.1 | Fix or quarantine the three unrelated controls that make the freshness monitor exit red on every run. | Freshness monitor exits 0 on a normal day for seven straight days. |
| 4.2 | Retire or repin the report-intake deployment guard, which is pinned to a release from August and only checks the Cloudflare worker. | The guard runs from the current release or is removed with its plist. |
| 4.3 | Recovery discipline for agents on this pipeline: no health claim without a fresh post-change readback, no clipboard or delivery claim from an exit code, and a new material recurrence gets its own incident note on day one. These go into the agent rulebook with the Sept 4 record as the fixture. | Rules staged with the incident as proof. |
| 4.4 | Sealed two-vendor review of the Sept 4 chain (round 6 was started but never completed), then formal close. | Both review files complete, incident record closed with the Sept 5 morning as literal proof. |
A. Dead-sender notice to recipients (item 2.4). Today the design is: agent-owned failures never reach you or the recipients; they land in a Woz queue. That is the queue that had no actor. My recommendation: keep failures out of your inbox for repairs, but if no Market Pulse exists by 10:45 CT, the five recipients get a one-line "delayed, being repaired" note from a separate job. Silence is what let three days go by. Your call: notice or keep silent.
B. Back-fill Sept 1 and Sept 2. Those two Market Pulse reports were never sent and the scheduler will not send them retroactively. Sept 3 went out today. Do you want Sept 1 and 2 sent as late reports, or skipped?