RL over a calibrated market-making baseline learns a constant, not a policy

Everyone says reinforcement learning is doing wonders in trading. I spent months putting it on my own market maker to find out. It did not work — and how it failed turned out to be the part worth writing down.

The setup

A bounded-residual RL overlay on a calibrated Guéant–Lehalle–Fernandez-Tapia market maker, over roughly 150 days of Bybit limit-order-book data, across five pre-registered arms. The action space was built so that a ≡ 0 reproduces the calibrated baseline to within $0.011 over the five-day evaluation window, which is what makes “beats the baseline” a well-posed claim rather than a comparison of two different systems.

The hypothesis, the predictions and the decision rule went into git before the first run, so the goalposts could not be quietly moved afterwards.

Every discriminating prediction failed

The design mistunes one parameter in opposite directions on two arms. If a policy is repairing miscalibration in its own baseline rather than trading, the learned action should flip sign with the injected error. If it is merely learning that one direction is better on this data, the sign is the same both ways. Opposite-signed pairs are what make the test discriminating.

None of it held. The reference arm, which was required to be null, came back at z = +2.78 — so under my own registered rule the study routes to uninterpretable rather than to a clean falsification. I reported it that way rather than reaching for a better-sounding outcome.

Then the result that actually stung

The checklist I had written for judging other people’s papers says: take the policy’s mean action, freeze it as a constant, rerun. If the constant matches the policy, nothing was learned. I had never run it properly on my own work — only on one dial of four.

So I ran it on all four dials, across all fifteen runs. The constant recovered 132% of the trained policy’s edge. It beat the policy in fourteen runs of fifteen, and in every arm.

The policy was outperformed by its own average. A one-line static configuration change earns more on the reference baseline than every policy trained there — +$2.18/day against +$1.15 — and what learning adds on top of the static change, +$0.24/day, sits inside the spread of the untrained controls.

And the cause is accounted for, not asserted

In the simulator the baseline has negative expected value per unit traded. So the only reliable gradient available to the policy is trade less.

The policies traded 10.8% less notional. A prediction with no free parameters — each baseline’s own loss rate multiplied by the volume its policy avoided — accounts for 71% of the measured edge, with r = +0.91 and an OLS slope of 0.94. The improvement was a volume reduction, not better execution.

A second finding I was not looking for

The headline numbers came from three seeds per arm. Re-running all five arms at twenty seeds — 85 additional training runs — showed three of the five arm means inflated by 1.8× to 2.8×, and the reference arm’s score fell from +2.78 to +0.95. It moved further than the width of the threshold it was being judged against.

Which also means the direction a policy learns on a dial is not identified at a single seed. I had previously read a positive size action as evidence that my production baseline was undersized. That reading reverses under replication, and the paper withdraws it.

Two checks I would now ask of any RL-for-market-making result

• The static-equivalent ablation. Fold the mean action into a constant and rerun. It is the cheapest experiment in the study, and it settled mine.
• Per-unit economics, next to the volume delta. If P&L improved while volume fell, show P&L per unit traded. Mine was flat — I had not made the strategy better, I had made it smaller.

Neither appears in any of the four representative papers I surveyed. None of them reports traded volume beside P&L at all, which means a P&L improvement that is really a volume reduction is indistinguishable from an edge in every number they publish.

Checkable, including the parts that do not flatter me

The data is public, with a verifier that re-derives every number in the paper from it — over 300 assertions, non-zero exit on any disagreement. The paper also lists ten defects found in my own work, seven of which produced a plausible wrong number and raised no error at all.

The caveat that matters: this is a simulator result and it is unverified live. The simulator’s markout is known to be too harsh by an amount comparable to one asset’s entire measured markout, so every conclusion here is drawn from differences on the same window rather than from absolute P&L.

Read it

• Paper — ssrn.com/abstract=7313940 (DOI 10.2139/ssrn.7313940)
• Data and verifier — doi.org/10.5281/zenodo.22013725 (CC BY)