A simulator cannot find what its fill model cannot express

A backtest is not a reporting tool. It is the instrument you use to decide which strategies get real money, and like any instrument it has a calibration. The uncomfortable part is that its errors are not random. A fill model that is slightly too generous is too generous in the same direction on every run, in every regime, for every configuration you test. You cannot average that away with more data, and more data is exactly what makes it feel trustworthy.

The failure mode that matters is not “the number was off by 20%”. It is that the simulator quietly answers a question you never asked it, and you read the answer as a finding. What follows is the list of things a market-making simulator has to model, the ones it structurally cannot, and what accuracy is actually achievable.

What counts as a fill

Everything begins here, and most of the damage happens here. A passive order rests at a price. Three different things can happen at that price, and only two of them are fills.

• An aggressor trade prints at your level. The queue ahead of you absorbs it first. You fill only on the overflow, and only for as much of your order as the overflow covers. This is a partial fill by nature.
• The book gaps through your price. A genuine cross-through fills you in full, because the market traded past where you were standing.
• The level clears without a trade. Everyone at that price cancelled. This is not a fill. Nobody traded with you.

That third case is the single most common fabrication in a homemade backtest, and it is seductive because it is so easy to implement: watch the best bid, and if it moves below your price, call it a fill. A model that fills on level-clears manufactures exactly the fills a market maker most wants — the ones where the price came to you and then left without anyone hitting you. It will show beautiful markout, because it is scoring you on trades that never happened. Any simulator that does this is not conservative-with-error; it is producing the wrong sign.

Queue position, and the cancel model nobody writes

If fills depend on the queue, then the queue needs a model, and the queue does not only shrink because of trades. It shrinks because other people cancel. This is the part that most implementations skip, and skipping it is not neutral.

Consider the two obvious shortcuts. If you only advance the queue on trades, orders never work their way forward, and everything under-fills — a passive strategy looks far less active than it is. If instead you let an order reach the front and stay pinned there, everything over-fills. Both are wrong, and they are wrong in opposite directions, which produces the nastiest possible symptom: at one particular order size the two errors cancel, and the model looks calibrated. It stays looking calibrated for as long as you never change the size.

The formulation that avoids a fudge factor is to ask, when a cancellation happens somewhere in the level, what is the probability it was ahead of you? If your queue position is Q within a level of displayed size L, that probability is Q/L. Freshly placed, you are at the back, Q equals L, the ratio is one, and every cancel counts as ahead of you. Near the front the ratio approaches zero and almost no cancels help you. It is correct at both ends by construction, for any order size, with no constant to tune per venue and no recalibration against live.

That last property is the one to insist on. A model with a free constant fitted to last month's live fills is not a model; it is a memory of last month.

Latency, and the orders that are not really there

An order is not live when your code creates it. It is live when the venue acknowledges it, and it stops being live some time after you send a cancel, not at the moment you decide to. A simulator that treats placement and cancellation as instantaneous gives your strategy a reaction time no real system has, and it does so precisely in the moments that matter, because those are the moments when everyone is trying to move at once.

The concrete expensive case is a fill that arrives after you sent the cancel but before the venue processed it. Those fills are toxic by construction: the strategy had already decided the quote was wrong. If your simulator cancels instantly, that entire category of loss is invisible to it, and it will systematically overstate every quote-management improvement you test.

Market impact: the limit you cannot engineer around

Everything above is a modelling problem with a good answer. This one is not.

A backtest replays a recording. The recording does not respond to you. If you rest a large bid, the tape still shows the same taker hitting the same price a second later, when in reality some of that taker's flow would have hit you — and having hit you, the price might not have travelled as far, and the next three events would have been different. The counterfactual is not in the data. It is not a matter of writing better code; the information required to reconstruct it was never recorded, because it never happened.

Two consequences follow, and both are permanent:

• A replay is optimistic at size. You get filled without ever deterring, absorbing, or moving anything.
• A replay cannot answer capacity. “Does this edge survive at ten times the size?” is exactly the question whose answer lives in the part of reality the tape omits.

This is worth saying plainly because capacity is usually the question with the most money attached, and the simulator will always be willing to answer it.

The scaling trap

Here is a trap I would rank as the most dangerous one on this list, because it disguises an assumption as a result.

Ask a simulator how profitability scales with order size and it will tell you. But how it scales is frequently decided by the fill model's structure rather than by anything in the data. If order size enters only where the fill quantity is capped by available overflow, volume will scale close to linearly with your order size — not because the market works that way, but because that is the only place size was allowed to matter.

The number that made this vivid for us: correcting a queue model changed how a 3x order-size increase behaved in simulation by more than an order of magnitude in traded volume. Same tape, same strategy, same period. The old model and the new model disagreed wildly about scaling, and both had looked fine at the size they were built around.

There is a subtler version. If benign fills are capped by print size while cross-throughs fill in full, then as order size grows the toxic path keeps scaling while the benign path saturates — so the model has a built-in view that bigger is worse, independent of evidence. The rule that falls out of this: never cite your own simulator's scaling behaviour as evidence about scaling. Go and read what your fill model does with size before you believe any capacity curve it draws.

Fees, rebates, and the tier feedback loop

Fees are the easy part: they are a known function of notional, and any simulator that omits them is not worth running. Maker rebates are arithmetically just as easy and psychologically much harder, because a rebate turns a losing per-fill edge into a break-even one and it is tempting to stop the analysis there.

The part that genuinely does not fit inside a backtest is the tier. Rebate levels are a function of your own trailing volume, so your fee schedule is an output of the strategy, not an input to it. A simulation of a single configuration over a fixed window has no way to close that loop. Worse, if a tier changes during a live experiment, per-unit economics improve for reasons that have nothing to do with the change you were testing, and it presents as a clean result. Any experiment run across a tier boundary should be treated as confounded until proven otherwise.

Execution algorithms are easier to simulate than market making

These are not equally hard problems, and it is worth being explicit about why, because the same team often owns both and applies the same confidence to each.

When you simulate an execution algorithm, you are mostly consuming liquidity that is visibly present in the recording. The book you are about to cross is right there. Your own impact is the main unknown, and impact against displayed depth is a well-studied, parameterisable thing that can be fitted from your own historical executions. The counterfactual sits close to the observed world: the liquidity you take was really there, and slippage against arrival price is measurable after the fact on every single order you have ever sent. That is a tight feedback loop, and it supports genuinely good accuracy.

When you simulate market making, you are a passive participant whose presence changes which trades occur at all. Every fill is conditional on someone choosing to trade with you instead of with the person who actually got that trade in the recording. There is no observable ground truth for that decision, and no amount of historical data contains it. The instinct that market-making simulation is the less precise of the two is correct, and it is correct for a structural reason rather than an engineering one.

So what is good accuracy?

The first move is to stop asking the simulator for the wrong number.

Absolute PnL is not a target. For a market maker it is the sum of many small quantities with opposing signs, each carrying the model error above. Expecting a simulator to land it is expecting the impossible, and teams that judge simulators this way end up tuning the simulator until it reproduces a number they already believe.

What a market-making simulator can plausibly deliver, roughly in order of how much confidence it deserves:

• Rank ordering of configurations. This is the whole job. If the simulator says A beats B and live says B beats A, every other accuracy statistic is irrelevant. Conversely a simulator with large absolute error but stable ranking is a perfectly good instrument, because ranking is what you actually use it for.
• Fill rate and traded volume — same order of magnitude, ideally within tens of percent. These are directly checkable against live on the same window and are the fastest way to catch a broken queue model.
• Markout distribution — the right sign and roughly the right magnitude, and more importantly the right shape. The tail is where adverse selection lives, and a model that gets the mean right by averaging a missing tail against a fabricated one is not calibrated.
• Direction of response to a parameter change. If widening quotes reduces fills in live, it must reduce fills in simulation.

A reasonable working standard: fill rate and volume within a factor of two, markout sign correct with the tail present, and rank ordering that survives out-of-sample against live. If you have that, you have an instrument. If you have beautiful PnL agreement and unstable ranking, you have a curve fit.

Validate against live, or admit it is a prior

The only test that means anything is running the same configuration live and in simulation over the same window, and comparing fill rate, volume and markout — not PnL. It is unglamorous, it consumes real capital, and there is no substitute for it. A simulator that has never been checked this way is not a measurement device. It is a formal statement of what you already assumed, executed very quickly and rendered as a chart.

The discipline that follows is simple to state and hard to hold. Before believing any result, ask which component of the fill model produced it, and whether that component was measured or chosen. A surprising number of “findings” turn out, on inspection, to be a line of code someone wrote to keep the volume numbers from exploding.