The phantom queue: how our fill model lost 80% of its fills

We ran a market-making simulator against a live book for months. It under-filled by a factor of five, and its fills were biased toward the losing ones. Nothing crashed, no test failed, and every backtest it produced looked plausible. This is a note on the mechanism, on why it was invisible, and on the one piece of infrastructure that eventually made it obvious.

For context on what follows: the strategy is a paired maker. It quotes both sides of the book on two instruments for the same asset — the spot market and the perpetual future — and holds them against each other so the combined position stays close to flat. Long spot, short perp, roughly offsetting. Those two books are the spot leg and the perp leg, and they behave differently enough that the same bug hit them by different amounts. That turned out to be the most useful clue we had.

Books drain by cancellation, not by trading

A resting limit order sits in a queue at its price level. It cannot fill until the volume ahead of it is gone. So the question a fill model has to answer is: what makes the queue ahead of you shrink?

The intuitive answer is trading. Someone hits the level, the orders ahead of you get consumed, you move forward. That is real, and it is a small minority of what actually happens. On one instrument we counted 16,397 cancellations against 217 fills at our levels — roughly seventy-five to one. Modern books churn. Quotes are placed and pulled continuously by participants reacting to information, and most of the queue ahead of you disappears without a single trade printing.

Our model advanced queue position only on trade prints. The cancellation path existed, but was scaled by a constant that shipped defaulted to zero and was set in no configuration file. So in every simulation we ran, cancels drained nothing. Orders sat behind a queue that, in the simulated world, essentially never cleared.

Why this is worse than under-filling

If a model simply produced fewer fills than reality, you would have a sample-size problem — annoying, but statistically honest. This is not that.

With the queue frozen, the only remaining way to get filled is for a trade to sweep through your price with enough size to consume everything ahead of you and still reach you. That is not a random subset of fills. That is precisely the case where the market is moving against you decisively — a large aggressor, a dislocation, a sweep. The benign fills, where someone lifts your quote and the price goes nowhere, require you to be near the front of the queue, and you never got there.

So the model did not give us a noisy estimate of our edge. It gave us a systematically negative one, assembled from the toxic tail of the fill distribution. More tape would not have fixed it. It would have produced a tighter confidence interval around the wrong number.

The two errors that hid each other

The reason this survived so long is that it was not one bug but two, pointing in opposite directions.

Orders rarely reached the front of the queue, so small orders under-filled. But an order that did reach the front stayed there: queue-ahead was pinned at zero and never re-grew, so it camped at the front indefinitely and large orders over-filled.

Somewhere between those two regimes there is a clip size where the errors cancel. Our production configuration happened to sit near it. Every sanity check we ran — do simulated volumes look like live volumes? — was run at that clip, and passed.

The tell, once we looked for it, was a sensitivity that made no physical sense: a 3× change in order size moved simulated volume 32× ($304 at one clip, $9,598 at three times that clip, against $5,298 live). No real market-making book behaves that way. A calibration that is only correct at one point on a curve is not a calibration; it is a coincidence.

The fix needs no constant

The original design tried to answer “what fraction of cancels were ahead of me?” with a tunable constant. That is not a model. It is a per-configuration fudge factor, and it needs re-fitting for every instrument, every size, and every regime — which is exactly why it ended up defaulted to zero and never set.

The question has an answer that requires no tuning. If a cancellation occurs somewhere in a level of total size L, and you have Q units ahead of you, then that cancel was ahead of you with probability Q/L. So:

queue_ahead -= cancelled × (queue_ahead / level_size)

Freshly placed, you are at the back: Q = L, the ratio is one, and every cancel counts as ahead of you. Near the front, the ratio goes to zero and almost none do. It is correct at both ends by construction, for any order size, on any instrument, with no recalibration.

Note what deliberately does not appear: order size. Your queue position depends on when you arrived, not how large you are. Size enters once, at the point where overflow becomes a fill — which is what makes simulated volume scale roughly linearly with clip instead of exploding.

The bug underneath the bug

There is a detail here worth more than the modelling. For a period, this cancellation logic was inert even when enabled, and for a reason that has nothing to do with market microstructure.

Tape prices arrive as floating-point. Order prices are quantised to the venue's tick grid. The code compared them directly. For any price not exactly representable in binary — 0.2013, 61.12, 245.7, 1.0512, which is to say nearly all of them — that comparison is false. The level lookup returned “no match,” and the entire cancel drain was skipped silently.

A float comparison invalidated more results than every modelling decision in this article combined.

What actually found it

Not code review, and not tests. The tests passed; they encoded the same assumption as the model.

What found it was running a paper simulation in parallel with the live strategy — same book, same configuration, same window, both recorded per instrument and per leg. That removes every confound. It is not a backtest against history and a live run in a different regime; it is the same minutes, the same order book, one real and one simulated. Any divergence is model error, and it is a number rather than an argument.

That comparison said: five times fewer fills, and markout two basis points more pessimistic. Both, it turned out, from a single cause.

One caution learned the expensive way: match on both fill count and fill quality. A parameter that reproduces the live fill count while getting markout wrong is filling at the wrong moments. That is more dangerous than under-filling, because it looks calibrated.

The second bug, found by the same method

The cancellation fix closed most of the gap on the perp leg and barely moved the spot leg. That asymmetry was the clue: whatever remained had to be something the two legs experience differently.

They do. The spot leg quotes considerably further from the touch than the perp leg does. And the code that seeds a new order's queue position did this:

if price > best_bid  { 0 }          // improving the bid — you are alone
else                 { best_bid_size }  // ANY price at or below the touch

Any resting bid at or below the best bid was seeded with the size at the top of the book, however far below it actually sat. Quote near the touch and that is roughly right. Quote well outside it, where the real level is nearly empty, and you are charged the entire queue depth of a level you are nowhere near.

The error grows with quoting distance — so it hit the leg that quotes furthest out hardest, which is exactly the leg that stayed broken. Seeding from the actual level at our own price took the spot under-fill from 4.9× to 1.9× on one instrument and 2.3× to 1.8× on another.

The third bug was our own measurement

With fill counts close to live on one instrument, its markout still looked two basis points too pessimistic. That is the frightening outcome — right number of fills, wrong fills — so we went looking.

The gap was not real. Markout is quantised by the venue tick, and on that instrument one tick is 1.45 bps of the price. Every median we computed landed on a half-tick multiple, and jumped a whole step as the distribution crossed the 50th percentile. The simulator's own confidence interval on that median was [-2.17, -0.74] — it spanned the entire apparent discrepancy. Measured by the mean, which does not quantise, the same comparison was 0.2 to 0.6 bps across every horizon.

So the model was already agreeing with reality and our statistic was hiding it. Worth stating plainly: on a wide-tick instrument, a median markout is a step function of your sample, and comparing two of them tells you very little.

The bug that was not in the fill model at all

One instrument now matched live on both axes — fill count within ten percent, mean markout within a few tenths of a basis point. Another stubbornly under-filled by roughly a factor of two, and no fill-model change touched it: not the cancellation fix, not the seeding fix, not the queue floor, not four different treatments of the censored region, not the latency gate.

Every one of those measured fills. Eventually we measured placements.

live:  339.8 quotes/hour
sim:    65.6 quotes/hour

The simulated strategy was never putting orders in the book at the live rate. No fill model can fix that, because the orders it would have filled were never placed.

The reason is worth more than the bug. In that 24-hour window the live bot suppressed 99.6% of its own quoting decisions — two million of them — on risk gates. Three quarters came from a drawdown ladder keyed to cumulative session PnL. Another fifth came from an account-level gate: total gross exposure across every instrument the desk trades.

A per-instrument backtest cannot know that number. It is not a bug we can fix; it is a fact about what a single-instrument simulation is. The live system was quoting less than the simulation because it could see risk the simulation had no way to see.

So the honest statement is not “our simulator under-fills.” It is: simulated volume is a lower bound, not an estimate — and comparing simulated fills to live fills without first checking that both systems are quoting at the same rate is comparing two different regimes and calling the difference a model error.

What to take from it

• Ask what makes your simulated queue advance. If the answer is only “trades,” your fill sample is biased toward the fills that hurt.
• Test parameter sensitivity, not just the operating point. A 32× volume response to a 3× size change is a defect signature, and it is visible without any ground truth at all.
• Two errors in opposite directions will cancel somewhere. If your only validation is at that point, you have validated nothing.
• If a bug hits two parts of your system by different amounts, that asymmetry names the mechanism. Ours differed by quoting distance, which pointed straight at the queue seeding.
• Check your statistic before you trust a discrepancy. A median on a tick-quantised quantity is a step function; ours invented a two basis point gap that the confidence interval alone would have disproved.
• Before comparing fills, compare placements. If the two systems are not quoting at the same rate, every fill-side number is measuring that difference instead of your model.
• Run paper alongside live. It is the only comparison with no confound, and it converts “is our simulator any good?” from a debate into a measurement you can track per release.