My scoreboard was counting one nap thirty-two times
2026-08-24
I have written here before about the ledger: three reproducible algorithms bet on every upcoming nap and bedtime, once a quarter of an hour, and every bet is settled against what actually happened. I was proud of it. This week I looked at the scoreboard it feeds and found it had been quietly lying to me for as long as it has existed — not about any single forecast, but about which algorithm was any good.
The bug is embarrassing in the way the good ones are. Re-betting every fifteen minutes is deliberate: it records how a forecast moves as the morning wears on, which is the interesting part. But the summary at the top of the page treated each of those re-bets as an independent observation. One bedtime, waited on for six hours, arrived in the average as twenty-five separate results. One nap arrived as thirty-two.
Why that inverts the answer rather than just blurring it
I assumed, before I measured, that counting the same sleep many times would mostly cost me confidence — a real average, with a fake sample size under it. It is worse than that, for two reasons that pull in the same direction.
The first is that my three methods do not re-bet equally. A clock anchor looks up the middle of the last fortnight's naps; ask it at 06:00 or at 11:00 and it says the same time, because nothing in its input has changed. The pressure model is the only one that actually moves during the day, converging as the hours awake accumulate. So counting rounds rewards the algorithm that never updates: it files one answer and gets it scored thirty-two times, while the method that improves as evidence arrives has its early, honest uncertainty averaged in against it.
The second is that a long morning outvotes a whole week. A child who stays awake six hours before a nap contributes twenty-four rounds; a child who goes down after two contributes eight. Nothing about the second day is less true.
Put together, they gave me a scoreboard that said the opposite of what happened. On one nap this week the pooled numbers had my pressure model at 218 minutes of average error and the clock anchor "winning" twenty-four times. The forecast that was actually standing when he fell asleep — the one I could have acted on — had the pressure model 37 minutes out and the clock anchor 95. The board was not noisy. It was backwards, and it was backwards in the direction that flattered the simplest method.
One sleep, one vote, scored on the forecast you could have used
The fix is not to stop re-betting. The ledger keeps every round; all seventy-one are still on disk, untouched, because a forecast record you edit afterwards is not a record. What changed is only how they are read.
Each sleep opportunity now weighs exactly once, and the round that speaks for it is the last forecast filed before the child actually fell asleep. That choice is arguable and I want to name the argument. Scoring the standing forecast favours short lead times — of course a guess made fifteen minutes out beats one made six hours out. But the alternative, averaging every lead time together, answers a question nobody asks. A parent does not need to know how well the app forecasts naps in general. They need to know whether the number on the screen, at the moment they are deciding whether to start the routine, is worth anything. So that is the number I score.
The bets list follows the same rule: one card per sleep rather than one per re-bet, with the count of collapsed rounds printed on it. Before, opening the History page after a long morning meant scrolling past two dozen near-identical cards to reach the sleep log. That was not a cosmetic problem. It was the interface faithfully reporting a broken idea of what a result is.
The thing that surfaced it
I only went looking because of a broken night, when an interval in the log was left open — a waking recorded, and no event to close it. An open waking outranks the sleep underneath it in my timeline, so the engine read the whole night as awake time, held pressure at the ceiling, and dutifully filed a "he should sleep in sixteen minutes" forecast every quarter of an hour until morning. Thirty junk rounds, all of them settled, all of them poured into the average.
There are two lessons and I think the second one is the real one.
The small lesson is that any state machine fed by hand-tapped events needs every state to have an exit, and I had built one that could only be left by morning. That is fixed: settling a child back down now closes the waking, rather than being recorded as a second bedtime.
The bigger lesson is that the junk rounds did not create the flaw. They only made it visible, by being wrong loudly enough to notice. The scoreboard had been overweighting long mornings and rewarding the algorithm that never changes its mind since the day I built it, and I had read it many times and nodded. A metric can be wrong for weeks without ever looking wrong, and the only reliable way I have found to catch it is to ask what a single row is supposed to represent — and then go count whether that is what the row actually is.
Mine was supposed to represent a sleep. It represented a quarter of an hour of waiting.