The biggest number is always the bug
Nine times in one research programme, the largest contributor to a good-looking result turned out to be a defect. Not a rounding error: the single row, fill or market that carried the whole finding. Three disciplines survived the experience, and one of them is just a rule about which row to read first.
Over one research programme on prediction markets, nine separate results looked good and then did not. In each case the failure had the same shape: the largest single contributor to the number was a defect. Not noise, not a marginal misestimate. One fill, one row, one market, one parsed string, carrying the finding on its own.
Nine is enough to stop calling it bad luck and start calling it a prior. This is the list, the pattern underneath it, and the three habits that survived.
The nine
| What it looked like | What it actually was |
|---|---|
| A clear negative markout on the first marking run | One fill more than twice the size of the quote it was matched against carried essentially all of it. The marker credited the whole residual after the queue and never capped at our own size, because the decision row logged the quote's price but not its size |
| A spread bucket that "excludes zero" and is positive | Seven fills across four tokens, one token contributing four of them. The bootstrap treats clustered fills as independent |
| "The venue's mid-band books are quoted at one tick" | Measured on the top few hundred events by volume, the most contested 1% of the venue. The whole-universe median was above the cost of waiting, not below it |
| A quoting engine finding nothing to quote, for weeks | The engine and the recorder were selecting markets independently. The disagreement looked exactly like a null result. Once one published and the other consumed, quoting went from 0% to 44% |
| A strategy losing 1.7 c/share, close to break-even | Markout marks to mid and silently assumes a costless unwind. Both exits actually available cost about 6.7c. The headline was flattering by four |
| Ten of twelve resolved markets returning "no such market" | The query excludes closed markets by default and succeeds with an empty list. An API filter was read as a market attribute, and it had already caused one finding to be retracted as an "artifact" |
| A product line apparently discontinued | We queried for a field of that name. It is a top-level request type. Asking correctly returned live markets |
| A large, persistent mispricing between two related contracts | Question-text semantics. "Between $64k and $66k" is a terminal condition, "reach $66k" is a barrier. And "by 6:30 PM on August 10" binds the deadline to a time, not to the day. Different payoffs, compared as if identical |
| A strong edge concentrated in one token | The strike parser had no decimal group, so several distinct strikes all collapsed onto the same integer and were compared against each other |
The pattern
These are not nine kinds of mistake. They are three, and it is worth separating them because they need different defences.
Four were semantic. The data was correct and we asked it the wrong question: a barrier compared to a terminal condition, a deadline attached to the wrong unit, strikes collapsed by a regex, a handicap that was not scoped to the match it appeared to describe. Nothing in the pipeline can catch these, because every number involved is real. They are caught by reading the underlying question text, which is an act of manual attention, not a test.
Three were silent-empty failures. A query that succeeds and returns nothing is the most dangerous response an API can give, because an empty result reads as a fact about the world. "No such market" and "this filter excluded it" are indistinguishable at the call site and completely different in meaning. Both of ours turned into published claims: one retracted finding and one nearly-published assertion that a product had been discontinued.
Two were statistical concentration. A bootstrap interval that excludes zero is a strong-looking object, and it is computed on the assumption that observations are independent. Fills cluster by token, by minute, and by dislocation. One market repricing during one bad minute produces several correlated fills that the interval counts as several independent pieces of evidence.
The common factor across all three groups is not carelessness. It is that every one of these defects made the result better. A defect that makes a result worse gets investigated immediately, because a disappointing number is uncomfortable and someone goes looking. A defect that makes a result better gets written up. The asymmetry is in the reviewer, not in the code.
What survived
1. Rank the output and interrogate the top row first, on the prior that it is broken
This is the cheapest of the three and it caught the most. Every one of the semantic failures above was found by reading the top hit's underlying text, or by checking one number against a quantity it had to be consistent with. Not one was found by the result looking wrong, because they all looked right. That is the whole problem.
A screen whose largest hit has not been personally interrogated has not been run.
The operational version: before reporting an aggregate, sort by contribution and ask what happens to the conclusion if the top row is deleted. If the answer is "it disappears", you do not have a result, you have one observation with an interval drawn around it.
2. Pre-register the rule, including the predicted magnitude, before testing it
The strongest signal in our data was time of day: three consecutive significantly negative hours against a much smaller baseline. The temptation was to gate those hours out and report the improvement.
That would have been fitting noise. The rule was found in the same data that would score it, and with twelve hour buckets some look bad by chance. So we wrote the rule down first: the exact hours, the exact behaviour, everything else held constant, and a predicted magnitude with a predicted sign. Then we ran it on new data.
The prediction was right, including its warning that a successful gate would not make the strategy profitable. The gate removed 44% of the loss and left it clearly negative. That is a far more useful outcome than a larger improvement would have been, because it is believable.
The predicted magnitude is the part people skip, and it is the part that does the work. A pre-registered direction can be satisfied by almost any result. A pre-registered number cannot, and writing one down forces you to state your model before the data can flatter it. There is a corollary: at a longer horizon our gate looked significantly better, and we had pre-registered the shorter one. Promoting the horizon that worked, after seeing that it worked, is exactly what the discipline exists to prevent, so it went into the file as an observation needing its own confirmation.
3. State the selection rule immediately next to the conclusion
The most expensive error in the list was not a code defect. We measured spreads using a recorder subscribed to the top few hundred events by volume, and then wrote a sentence about the venue. That is the tightest, most contested slice of the market, where every professional maker already is. The whole-universe check reversed the conclusion.
What makes this one instructive is that the bias was already written down. The selection rule was documented, we had even noted that the recorder was biased against finding the markets we wanted, and the venue-wide claim got made anyway, because the rule lived in one file and the conclusion in another.
A conclusion inherits the selection of its sample.
The operational form is stricter than it sounds: the sentence stating the result must contain the sentence stating how the sample was chosen. Not a methods section, not a footnote, not a neighbouring file. If those two sentences can be read apart, eventually they will be.
There is an engineering version of the same rule, learned from the weeks the quoting engine spent finding nothing. Two programs that select independently will silently disagree, and the disagreement looks like a null result. When one component chooses a universe and another measures against it, the chooser must publish and the measurer must consume. Never re-derive.
Report n before direction
One last habit, small enough that it does not need a section of its own, and it is the one we would keep if forced to keep only one. State the sample size before stating the sign.
"Spread bucket 4 to 6c is positive and excludes zero" invites belief. "Seven fills across four tokens, one contributing four of them, positive and excluding zero" invites the correct amount of belief, which is none. The information content is identical. The order decides what a reader does with it, including when the reader is you, six weeks later, reaching for a number to quote.
None of this makes the underlying work better. It makes the reported work survivable, which matters more when the results are negative, because a negative result is the one thing a defect cannot manufacture by accident. Every bug in the list above pushed in the same direction.