The claims most likely to be wrong are the ones written in sentences

A study I recently published spent its length arguing that measured trading improvements are routinely something other than what they appear to be. While writing it I kept a list of every defect in my own tooling that materially changed an intermediate result or the stated justification for a design choice.

The list came to ten. The distribution is the part worth reporting.

Seven raised no error

Only one failed loudly — a configuration key collision that rendered a profile unloadable, because a runtime persistence path wrote a per-leg key colliding with one declared elsewhere. That is the good case. It stopped.

One reported the opposite of what happened: four completed runs were recorded as FAILED because the process handle was not retained and the exit code read as null. One was a liveness defect that produced no wrong number at all — progress was reported per day rather than per evaluation pass, so a working run went silent for about forty-two minutes and was indistinguishable from a hang.

The other seven were silent. No error, no warning, a plausible wrong number.

And five of those seven lived in prose

My verification effort was aimed at tables. Every table figure in the paper was recomputed from the artefacts by a script that exits non-zero on any disagreement — more than three hundred assertions. None of that machinery reads a sentence.

Four of the sentences it did not read:

• “Adjacent days differ by up to 19×.” An inherited figure, never checked against the data it described. Measured, it is 1.4×. Found by me, not by tooling, after every table figure had already been verified.
• “The live policy requotes by cancel-and-replace.” Inferred from an operation-count table and never checked against the gate column in the same file. It is false — the live maker requotes by amendment, and the cancels were defence gates withdrawing quotes. It propagated into three documents before a reader who knew the system objected.
• An 87% volume share borrowed from a different asset. Used to blend a per-unit markout, which made a headline calibration ratio 2.4× where the measured mix gives 1.9×.
• A single day’s medians presented as the sweep. A per-leg markout table showed one day, printed beside a three-day table with nothing marking the difference. The direction it showed did not survive the other days. A directional conclusion and a further claim built on top of it were both withdrawn. Found only when the reports were prepared for release, three audits after the table was written.

A fifth is the most instructive because the correction was also wrong: gate-suppressed placement attempts were counted as executed operations, producing a figure of “77 places”. The correction issued for it — roughly ten times fewer operations — was itself wrong. The true figure is about a hundred and ten times.

Why this is a systematic failure rather than carelessness

Consider what verification actually covers in a research codebase. Tests cover functions. Assertions cover computed values. Recomputation harnesses cover tables. All of it operates on things that have a machine-readable ground truth to compare against.

A sentence has no such thing. When you write “the system requotes by cancel-and-replace,” nothing recomputes it, nothing diffs it against the artefact it describes, and it survives every review that is looking at numbers. It then gets cited by the next document, and by the one after that.

So the claims most likely to be wrong in any technical document are precisely the ones written in sentences, because they are the ones no script recomputes. A paper about unverified claims surviving in prose reproduced that failure four times in its own text before catching it.

Three fixes that generalise

• Gate collation on a comparability stamp. One of the silent defects pooled superseded runs scored under different control settings, inflating an arm from three samples to four and moving its score materially. Runs should refuse to be compared unless they were produced under the same conditions, and the tooling should enforce that rather than the analyst remembering.
• Judge completion on the output artefact, not the exit status. An exit code is a claim about a run. The artefact is the run. Where the two disagree, believe the artefact.
• Make liveness a signal that cannot lie. Report progress against the unit of work that actually advances, not a coarser one. Otherwise a healthy process and a hung process emit the same thing, which is no signal at all.

The uncomfortable part

Three of the prose defects were found only after the paper was first considered finished — two by external review, one by a reader who knew the system well enough to recognise that a sentence describing it was false.

Which means the mechanism that caught them was not a process. It was somebody who already knew the answer reading the sentence carefully. That does not scale, and I do not have a better answer than writing the list down and checking prose claims against artefacts deliberately, the way one already checks numbers.

The full defect table, with the failure mode of each, is in the paper — ssrn.com/abstract=7313940, §9.