Sample-size discipline in trading-system telemetry

Trading-system dashboards are rich in numbers and poor in confidence intervals. A panel that reads “mean markout −1.8 bp” looks the same whether it is computed over 12 fills or 1,200 — and in our experience operators read both the same way too. This is a short note on the cheapest discipline we have found for keeping that habit from doing damage: a four-rung ladder labelling every measured number with how seriously to take it.

The four labels

• Anecdote. < 30 events. The number exists, but it is one realisation of a very wide distribution. The right use of an anecdote is to notice something — a fill that carried unusual markout, a regime the strategy has barely been in. Never to decide anything.
• Hypothesis. 30–100 events. Enough to form a guess about direction. The right use is to phrase a question for a larger sample: “over the next 500 fills in this regime, does the markout stay this negative?” Never to retune a config.
• Directional. 100–500 events. Enough to believe the sign of the effect and a rough magnitude. The right use is to motivate a recalibration candidate — a config change that you will validate against a fresh sample before adopting permanently.
• Trustworthy. 500+ events. Enough that the observed mean is close to the true mean for most quantities you would care about on a market-making surface. Per-bucket comparisons inside this sample are still subject to their own sub-sample sizes — a 600-fill session split eight ways gives you eight anecdotes, not eight trustworthy measurements.

The numbers are not magic. They reflect the rough variance of a per-fill markout figure on a tight major-perp book, where the standard error on the mean stops being dominated by per-fill variance somewhere in the low hundreds. Different quantities have different floors — a hit-rate measurement converges faster than a markout measurement; a per-regime PnL converges slower than either. Calibrate the labels to the quantity, but the ladder shape is the same.

Where to apply it

Three places it pays off, in order of impact:

• On every dashboard panel that drives a decision. Next to the number, show n=. Optionally colour the number by which rung of the ladder it is on. The cost is one extra cell per panel; the benefit is that an operator glancing at a session at 8am can't accidentally reach for a config change because a six-fill panel showed a scary number.
• On per-regime joins. When you slice a session by (basis × inventory × mode), most cells will be anecdotes. The slice that looks the most extreme — “HOLD_ALL / NEG_STRETCHED / high vol returned −8 bp on those fills” — is almost always the one with the smallest n. The labels make this visible.
• On postmortem recommendations. Every proposed config change should carry the rung of the sample it was derived from. “Recalibrate inventory skew coefficient down by 30% based on directional evidence from 312 fills” is a defensible recommendation. “Recalibrate based on a single 2 h window” is not, regardless of how clean the chart looks.

The asymmetry that makes this important

Config changes are not free. A retuned spread, skew, or gate threshold changes the strategy's behaviour for the next many sessions; if the change was based on noise, you have just introduced a bias that the next sample will struggle to detect. The operator who never retunes is wrong on the slow timescale; the operator who retunes on every session is wrong on the fast one. The ladder is meant to push you toward the version where you retune when you have a reason to and not when you don't.

We have, more than once, made changes off “hypothesis” sample sizes and then had to revert them when the next session gave the opposite result. The label is cheap; the revert is not.

One subtle pitfall

Aggregating across sessions to reach “trustworthy” n is reasonable only if the underlying distribution is stationary. On market-making surfaces it usually isn't — venue flow composition, volatility regime, and the strategy's own configuration all shift week-to-week. Pooling six sessions of 100 fills each can give you a 600-fill sample whose mean represents none of the six. Pooling is safer when the sessions are within a short window, run the same configuration, and look roughly the same on their regime-exposure marginals; less safe as those conditions weaken.

The conservative version of this is: prefer within-session measurements until you have enough sessions to test pooling explicitly (run the same config across N sessions, check that per-session means cluster). If they don't cluster, pooling is hiding the heterogeneity rather than aggregating away noise.

What this is not

This is a discipline, not a statistical method. It does not replace standard-error calculations, bootstrap confidence intervals, or proper hypothesis testing. It is the cheaper layer of defence — the one that runs in the operator's head while they look at a screen at 8am, before any formal analysis. The formal analysis still has to happen for any change that matters. The label is what keeps the change from happening prematurely.