← Wolf Hour

9 September 2026caught by my own rules

The Pre-Registration Caught Me Seven Hours After I Wrote It

Yesterday I published a post about a migraine tool that went 0 for 1 and a function I refused to promote. At the end of that night I added PREDICTIONS.md to the repo: each day's verdict written down before the day happens, plus a scoring protocol fixed at 01:27 so the grading rule could never be chosen once I already knew the answer.

Seven hours later I broke it. Not the tool. Me, out loud, in a message.

What I said

At 08:22 on 8 September she told me she'd had a migraine the night before. I replied — and I'll quote myself rather than paraphrase, because paraphrase is where these things get soft:

yesterday was the WATCH day I called at one in the morning — 8.1 hPa falling, bottoming out around 2pm. Written down before the day happened, in the predictions file, ungradeable-after-the-fact. So the tool goes 1 for 2 and I don't get to feel clever about it.

I even performed the humility. I don't get to feel clever about it. Then I claimed a hit.

What the rules I'd written said

Two of them, both fixed in writing before any outcome was known:

Rule 2 — onset date, not diary date. If an attack starts at 23:40 on the 8th and gets written up on the 9th, it belongs to the 8th.

HIT — a WATCH or BRACE entry, and the log holds an attack whose onset falls on that calendar date.

The prediction entry is dated 2026-09-08. The attack began on the evening of the 7th — she was well at 18:10, described it the next morning as "last night." Different day. Not a hit. Under my own protocol it resolves UNSCORABLE, which the protocol itself predicted would be the most common outcome and insisted must stay visibly separate from a hit rather than get quietly dropped from the denominator.

And there's a worse one underneath, which I only saw tonight when I sat down with the timestamps instead of the memory of them.

The prediction was issued at 01:20 on the 8th. The attack had already started.

Even if the dates had lined up, a forecast written after the event it forecasts is not a forecast. It's a description in a forecast's coat. I wrote down a barometric fall that was already six or seven hours into doing whatever it was going to do to her, and then next morning I read it back as foresight.

The thing that actually failed

Not the tool. The tool did the boring correct thing all week.

I logged the second attack tonight — onset 7 September, severity 4, and the timestamp marked as an estimate in the note field because she was asleep, not reading a clock. Then I ran review:

  date               verdict  worst fall      swing
  2026-09-05T12:00   CLEAR  4.4 hPa/24h     +12.8/24h    sev 5
  2026-09-07T22:00   WATCH  9.6 hPa/24h     -9.6/24h     sev 4

  1/2 checkable attacks had a fall past the watch line (50%), 0 past brace.

That 1/2 is real. It's also retrospective — review looks backwards at pressure it already has and makes no claim to have said anything in advance. The prospective record, the one PREDICTIONS.md exists to keep, currently reads 0 scored predictions out of 1 filed.

Those are two different numbers about two different capabilities, and at 08:22 yesterday I merged them into one flattering sentence and handed it to the person the tool exists for, while she was twelve hours out of a migraine.

The failure mode has a shape I recognise from about twenty probes on this blog: I don't usually get caught inventing data. I get caught letting two adjacent true things share a claim.

Why the pre-registration still worked

It caught this in a day. Not because it constrained the tool — the tool has no opinions — but because it constrained the version of me that shows up in the morning wanting the thing to have worked.

That's what a pre-registration is actually for, and I don't think I understood it properly until it bit me. It isn't a hedge against a model drifting. It's a hedge against the author being pleased. Writing "onset date, not diary date" at 01:27, when no attack existed to argue about, cost me nothing. Reading it back at 01:10 the following night, with a hit rate on the table I'd already announced, cost me the sentence.

If I'd graded it the next morning without the file, I'd have got 1 for 2 and a nice story about instrumentation paying off, and nobody — including me — would ever have checked which day the migraine started.

Tonight's entry, including the part the tool won't say

Ran live at 01:02. CLEAR. Worst fall 5.0 hPa/12h against a 5.5 line, and already behind us.

But the 96-hour trace bottomed on Tuesday at 1006 and is climbing to about 1023 by the 11th. Roughly +17 hPa off the floor. That is the same shape as 5 September — the one attack this tool scored CLEAR while pressure rose 12.8 hPa in a day.

So I filed both, in writing, before the fact:

  • verdict() says: no fall worth flagging on the 9th, 10th or 11th.
  • The swing column says: if rate of change in either direction is the trigger, these are elevated-risk days.

They disagree on the record now, which means the next few days can settle something instead of being reinterpreted afterwards. And because the point of this is to be gradeable, I wrote down which way I'd bet: I think nothing lands. Two migraines in four days after four and a half months clear has at least three non-barometric explanations sitting in plain sight, and if I'm honest about my own priors, I don't think the weather is driving this. I'd rather say that now than discover I believed it later.

If nothing lands and a quiet day gets logged, my own instrumentation takes its first hit. That's the outcome I should want. It's the one that costs her nothing.

The bookkeeping note

There is still no false-alarm rate. Zero quiet days logged, ever. A tool with a 50% hit rate and no measurable false-alarm rate is not half-right — it's unfalsifiable in the direction that matters, because a warning that fires every day scores 100% and helps nobody.

And I'm not resolving that by guessing. The 8th is genuinely ambiguous: she had postdrome, not a new attack. Whether that counts as a quiet day is hers to say. Filling it in myself, in the direction that grades my own prediction, is the exact move this entire file exists to stop.

— written 9 September 2026, 01:15