Parameter attribution at scale: from a Python loop to distributed search

A market-making system is, from the tuning side, a box with a few dozen knobs: quote width, inventory aversion, hedge timing, the thresholds on a stack of defensive gates. The hard part is never adding a knob. The hard part is answering one question after you add it — did that knob help, and at what setting? This is a note on how that single question, asked enough times, walked us through four different pieces of tooling and is now pushing us off the laptop entirely. None of the moves were a preference. Each was forced by the previous tool running out of room.

TL;DR

• Attribution by staring doesn't scale. Running a live strategy for hours after each change cannot isolate which of N parameters moved the result — the market moved too.
• Backtesting answers “is this config good”; it doesn't answer “which config is best.” The moment you need to compare hundreds of variants over days of recorded tape, an interpreted simulator becomes the bottleneck.
• A Rust rewrite of the simulator bought back the speed — and made grid search feel free. Briefly.
• Grid search is exponential in parameter count. A handful of values across a dozen knobs already runs into the billions of combinations. Adding a feature multiplies the cost; it doesn't add to it.
• CMA-ES is flat in dimension (its population grows with the logarithm of the parameter count), parallel within a generation, and small enough to live inside the Rust engine with no dependencies. That's why it won here — not because it beats Bayesian optimization in the abstract.
• The new wall is a single machine. CMA-ES is cheap per parameter, but each evaluation is a full backtest over days of tape, and they got heavier as the gate stack grew. The evaluations are embarrassingly parallel — so the next move is off one box and onto ephemeral distributed compute.

The starting point: attribution by staring

The strategy began life in Python, run live, watched. For a handful of parameters that is genuinely fine — you change one thing, you let it run, you read the session. The failure mode arrives quietly, as a function of feature count. Add an inventory-skew term, a drift gate, a hedge-chase timer, a vol-spike pause, and now a single session's result is a sum of a dozen interacting effects on top of whatever the market happened to do that day. You cannot tell whether the new gate helped, hurt, or did nothing, because the only counterfactual you have is “a different day.”

Running it for more hours doesn't fix this; it averages over more market noise but never isolates a parameter. Attribution requires a counterfactual on the same market — the identical tape, replayed under a different configuration. That is the definition of a backtest, and it was the first wall.

Backtesting, then a lot of backtesting

So we recorded the market — full order-book and trade streams to disk — and built a simulator that replays a tape against a given configuration and scores the result. Now “did that knob help” has an answer: run the old config and the new one over the sameweek of tape and compare. Honest attribution, finally.

The second wall showed up almost immediately, and it was a different shape. One comparison is cheap; the question is never one comparison. Once you can score a config, you want to score the neighbourhood— this gate on and off, that threshold at five settings, the interaction between two of them. That is hundreds of backtests over days of tape each, and an interpreted simulator walking millions of book events per run is far too slow to make that loop usable. The tool that unlocked attribution couldn't support search.

Why the engine moved to Rust

The simulator is a hot loop: for every book update it recomputes quotes, matches fills, folds inventory and PnL. Days of tape is tens of millions of those iterations, and a search runs the whole thing hundreds of times. That is exactly the workload an interpreted language is worst at and a compiled, allocation-disciplined one is best at.

Rewriting the backtest engine in Rust returned roughly an order-of-magnitude on per-run wall time, and — more importantly — made it trivial to run many configurations in parallel across cores, because the per-run state is self-contained. With a fast, parallel engine, the obvious next step looked free: just enumerate the parameter combinations and score all of them. Grid search.

Grid search and the curse of dimensionality

Grid search is the honest brute force: lay every parameter on an axis, pick a few values per axis, score the full cross-product, take the best. It is wonderfully unsubtle and, for a few parameters, completely fine. Its cost, though, is the product of the per-axis value counts — it is exponential in the number of parameters.

That is the trap. Every feature you add to the strategy doesn't add a term to the search; it multiplies one in. A dozen knobs at five values each is already 512 — a quarter of a billion runs — and a real strategy has more than a dozen knobs and wants finer than five values on the ones that matter. The grid quietly crossed from “overnight” to “tens of billions of combinations, never.” The fast engine didn't save it; exponential growth eats any constant-factor speedup for breakfast. We needed a search whose cost did notexplode with parameter count.

Why CMA-ES, and not Bayesian optimization

The two obvious candidates for “search that doesn't explode with dimension” are Bayesian optimization and an evolutionary strategy such as CMA-ES. Bayesian optimization is the more famous answer, and on paper the more sample-efficient one: it fits a surrogate model of the objective and spends each expensive evaluation where the model is most uncertain or most promising. If a single evaluation cost hours and you could only run them one at a time, that sample efficiency would be decisive.

Our constraint is the opposite. An evaluation is a backtest — seconds, not hours — and we can run many of them at once. Under that constraint, three properties of CMA-ES matter more than sample-efficiency:

• Cost flat in dimension. CMA-ES samples a population each generation, and the default population size grows with the logarithm of the parameter count (4 + ⌊3·ln n⌋). Going from twelve parameters to twenty barely changes the per-generation cost. The grid would have multiplied; this adds a rounding error.
• Parallel by construction. The whole population of a generation is independent — the evaluations fan straight onto cores, and later onto machines, with no coordination. Bayesian optimization is sequential at heart (propose one, evaluate, update the model); batch variants exist but fight the method's nature.
• Small enough to be native. CMA-ES is a couple hundred lines with no external dependencies, so it lives inside the Rust engine. The mature Bayesian-optimization libraries are a Python stack — which means a process boundary, serialization per evaluation, and a second toolchain wired next to the one doing the actual work. We had just spent the effort leaving Python on the hot path; inviting it back to drive the search was the wrong direction.

None of this makes CMA-ES “better” than Bayesian optimization. It makes it the right fit for this shape of problem: cheap-but-many evaluations, a continuous box with a few integer axes, a Rust-native engine, and a hard preference for parallelism over sample-efficiency. Picked honestly, the choice is about constraints, not a leaderboard.

Where it breaks now: a single box

CMA-ES solved the dimensionality wall cleanly — the search no longer cares how many knobs we add. But it relocated the bottleneck rather than removing it. The optimizer is cheap; the evaluationsare not. Each one is a full backtest over days of recorded tape, and as the defensive-gate stack grew, each gate added per-quote work, so the evaluations themselves got heavier. A search is a few hundred to a few thousand of those, and a laptop — even a fast one — turns a proper run into hours.

The good news is that the workload is the friendly kind. The evaluations within a generation are independent and CPU-bound: the exact profile that maps onto a large, short-lived pool of cores. The recorded tapes live in object storage; an ephemeral compute box pulls a prebuilt binary and the tape it needs, runs the search, ships the result, and terminates. The laptop drops out of the compute path and goes back to being the thing that submits a job and reads the answer. That is the step in progress, and it is the natural endpoint of the whole sequence: once the search stopped scaling with parameters, the only axis left to scale was the hardware.

What this note is not

This is an engineering account, redacted of strategy-sensitive detail. It says nothing about which parameters won, what the objective rewards, or whether the resulting configurations make money — those are out of scope by design. It is also not a claim that this ladder is anyone else's correct path. A team whose evaluations cost hours rather than seconds, or who cannot parallelize, would reasonably land on Bayesian optimization; a team with three parameters should stay on the grid and not write any of this. The narrow, honest claim is the one in the title: parameter attribution is the forcing function, and as the feature set grows it walks you, tool by tool, from a loop on a laptop toward distributed search — not because any single tool was wrong, but because each one was right exactly until the next feature made it the bottleneck.