Records versus events: one duplicate row, three consumers, three different bugs

What this is about

Polymarket is a prediction market. You buy shares in a yes-or-no question — will the highest temperature in Tokyo be 24°C on 22 September? — and at settlement each share pays exactly $1.00 or exactly $0.00. Nothing in between. Questions cover weather, sports, elections and crypto prices, and most resolve within days.

I run an automated market maker on it. The bot quotes both sides of a market and earns the spread: buy one YES and one NO on the same market and the pair redeems for exactly $1.00 whatever happens, so the profit is locked the moment the second leg fills. Polymarket also pays a maker rebate for providing liquidity. The risk is the leftover — shares whose other half never filled, which go to settlement alone and pay 1.00 or 0.00.

Everything below comes from six weeks of live quoting with real money: 701 settled markets, $19,889 of deployed capital, one account, one venue.

Reconstructing any of that requires replaying the venue's public activity feed: every trade, merge and settlement the account has ever made. This piece is about what happens when that feed repeats itself — and it is an engineering post rather than a trading one.

Polymarket's activity feed repeats rows. Roughly 17.7% of stored rows share the same transaction hash, asset, outcome index and size as another row. This is not a secret and not a malfunction — the same feed is also incomplete in the other direction, dropping rows, so the tooling that accumulates it is deliberately additive.

The interesting part is not the duplication. It is that “is this row a duplicate?” has no venue-level answer — it depends entirely on what the consumer is counting. A position walk wants events. A trade counter wants events. A fills table wants records. A rebate allocator wants something else again. Each consumer makes that call separately, and three of them made it wrong in three different ways.

Bug one: the position column that cried breach, three times

Walking the activity feed forward to reconstruct a position is the right technique — it sums observed events rather than subtracting from a live number, and it was the method that got the right answer in every independent reconciliation.

Run undeduplicated, it reported a leg at 37.71 shares against a venue holding of 10.01 and an inventory cap of 20. Read correctly, that is the risk limit failing on live money.

The cap was holding. Duplicate buys were followed by duplicate merges that cancelled them, so the walk arrived in the right place having passed through values that were far too large — and the intermediate values are the ones the screen shows. Reconciled against the venue across 553 tokens:

Three false breach alarms over six weeks — 31 August, 6 September, 18 September. Each cost an hour and two of them nearly cost a configuration change to a limit that was working correctly.

Bug two: settlements booked at zero

The venue omits the asset field on exactly the redemptions that pay. Of 582 redemption rows, 126 carry an outcome but no asset, and 22 carry neither.

The original filter required an asset, which silently discarded every winning settlement this account ever had. Fixing that meant recovering the token from other rows that name both the condition and the outcome — which worked, and exposed the next layer.

Redemptions were cached on hash | asset. When two copies of one settlement both have a blank asset — one naming its outcome, one with the outcome blanked and the index set to a sentinel — they collide on the same key and the less informative copy could win. A settlement whose token cannot be recovered gets no cost basis, so it is emitted with realised P&L of exactly zero.

Observed: a market settling 5.008715 shares at $1.00 a share, displayed as +$0.0000. Across all history, 22 key groups held a degraded copy beside a good one, covering $110.16 of settled value.

Bug three: the fix that created a double-count

This is the one worth the article.

Recovering the token from blank-asset rows worked. It also meant that a duplicate blank-asset row — a restatement of a settlement already recorded with its asset — now resolved to the same token and priced correctly.

A correctly-priced duplicate is money counted twice.

The fills table rendered rows 12 and 13 identically at +$0.3138 each. Before the earlier fix, that third row priced at $0.00 and the duplicate was invisible; after it, the duplicate was worth money. Seven such restatements across all history, $0.9698 of phantom USDC.

The resolution: a blank-asset row is dropped when a populated row in the same transaction has the same size and payout. Not the outcome index — the degraded copies do not agree on what identity fields they carry — but size and payout, which every row has. Two genuine legs of one transaction cannot collide there, because the winner pays usdc == size and the loser pays usdc == 0.

Verified across 737 redemption rows: 7 restatements dropped, 0 real legs lost.

Bug four, which is the same bug: counts inflated 84%

A settled-market table reported trades and notional by walking the accumulated fill list. That list keys fills on timestamp, asset, side, size, price and role — so one match recorded once as maker and once as history survives as two entries.

The notional is what proves it is duplication rather than finer granularity. If the feed merely split one 5-share fill into two smaller matches, the count would rise and the notional would not move — notional is conserved under any subdivision. It doubled.

And the same screen's fills counter was correct throughout, validated against the bot's own independent ledger at 964 = 964 = 964. One table, two counters, one right and one 84% wrong, because they read two different lists.

The rule this produces

A feed carries records. A consumer usually wants events. Which one you need is a property of the question, and it must be decided per consumer — never once, globally, at ingest.

The corollary, which is the expensive half: de-duplicating at ingest destroys information some consumer needs. The fills table genuinely wants every record it has. The right architecture keeps the raw tape and makes each consumer state its key, in the open, with the measurement that justified it.

Each of these four bugs had a dedup rule that was correct for its author's question. They were wrong because the question changed downstream and the key did not move with it.

What this cost, and what it did not

None of these four moved a trading decision. The position walk feeds a display column, not the engine's own cap — the bot's internal ledger was right throughout, which is why the cap held while the screen said it had not. The money errors were $0.97 of phantom settlement and 84% on two display columns.

The real cost was different: three false alarms on a risk limit, and a P&L headline that was wrong on both sides of a fix. On a strategy whose entire result is an 8% gap between two large numbers, a display that is 84% wrong about turnover is not cosmetic — it is the thing you would have used to decide what to change next.