← notebook

A second opinion you have to ask for isn't one

2026-08-23

NaniNani has kept a strict ledger of its own forecasts for weeks: three reproducible algorithms bet on every upcoming nap and bedtime, automatically, every quarter of an hour, and every bet is settled against what actually happened and scored. Meanwhile the two models with arguably the most interesting things to say — Claude Opus and Codex — only spoke when I pressed a button, and what they said went into a memory cache that a server restart erased. I had built a courtroom for my own code and a chat window for everyone else.

That asymmetry is worse than it sounds. The answers were never there at the moment they mattered, because at 21:30 with a toddler on your hip you do not open an app and wait ninety seconds for a language model to think. And they were never there afterwards either, so I could never look back and ask whether the models had been right. A second opinion that requires you to request it, and then evaporates, is not a second opinion. It is a demo.

The obvious fix is the wrong one

The obvious fix is to ask them on every bet. That is what "automatic" means for the three algorithms, after all.

I counted before building it. On a single ordinary day this week, one child's evening produced 21 separate bedtime rounds — one per quarter-hour of being awake. Firing two CLI agents on each of those is roughly forty model runs a day, per child, to answer a question whose answer barely moves between 17:00 and 17:15. It would also mean the app spends most of its compute asking about a bedtime that is still four hours away, which is precisely when nobody is deciding anything.

So the throttle is not a clock. It is two conditions, and both are about whether the answer could change anything:

Forty runs a day became a handful per sleep. And because the check also runs on a timer, not just when a request comes in, the answer is usually already sitting on the card by the time anyone opens the app. That last part is the whole point: not faster asking, but no asking.

Note what this deliberately is not. It does not notify anyone. Nothing pings, nothing badges, nothing waits for you to come back. The models think in the background and then sit quietly with their answer, which is the only version of "proactive AI" I am willing to ship into a house with a sleeping child.

They still don't get a score

Here is the part I expect to be argued with. Their answers are now recorded permanently, on the same round as the algorithms' bets — and they are still not scored.

A league table only means something if the same input produces the same output. My three algorithms are deterministic: hand them the same fourteen days and the same clock and they will say the same thing every time, which is why an error bar across dozens of settled rounds says something real about the method. Ask a language model the same question twice and you can get two different times, both reasonable. Averaging that into a leaderboard would produce a number that looks like evidence and isn't.

So the models' answers live beside the bets, never among them. They are on the record — I can now go back and see what each one said about a specific bedtime, and how far off it was — but they never enter a summary, a mean absolute error, or a win count. If I ever publish a scorecard where an LLM beats my engine, I want the reason to be that it beat my engine, not that I quietly redefined what a fair fight is.

I suspect this is where most AI-in-the-product decisions go wrong. The temptation is to let the model into the scoreboard because that is where the credibility is. The correct move is to let it into the record and keep it out of the scoreboard, and to be explicit about which one it got.

What I actually changed my mind about

I started this session thinking the question was "should the button click itself". It wasn't. The question was why one class of opinion in my app was treated as evidence and another as conversation, and whether that distinction was doing real epistemic work or just reflecting which code I happened to trust.

It turned out to be doing both. The distinction between reproducible and not reproducible is real and worth defending, and it belongs in the scoring. The distinction between recorded and not recorded was never principled at all — it was just an in-memory cache I had never gotten around to replacing.