A gap must be a row: 4.9 million frames, zero dropped connections, and why you record anyway

What this is about

Polymarket is a prediction market. You buy shares in a yes-or-no question — will the highest temperature in Tokyo be 24°C on 22 September? — and at settlement each share pays exactly $1.00 or exactly $0.00. Nothing in between. Questions cover weather, sports, elections and crypto prices, and most resolve within days.

I run an automated market maker on it. The bot quotes both sides of a market and earns the spread: buy one YES and one NO on the same market and the pair redeems for exactly $1.00 whatever happens, so the profit is locked the moment the second leg fills. Polymarket also pays a maker rebate for providing liquidity. The risk is the leftover — shares whose other half never filled, which go to settlement alone and pay 1.00 or 0.00.

Everything below comes from six weeks of live quoting with real money: 701 settled markets, $19,889 of deployed capital, one account, one venue.

None of that analysis is possible without market data, and prediction-market data has to be recorded rather than requested: the venue publishes a live order-book stream and no historical endpoint. This piece is about building a recorder for that stream — three rules, what the venue turns out to publish that nobody subscribes to, and a negative result that exists only because of rule one.

You can buy Polymarket book history. Tardis covers it from May 2026; a handful of specialists — Predexon, Telonex, PMData, PolyHistorical, TickFoundry — sell tick-level order books from between August 2025 and mid-2026, at $0 to $499 a month. If you need history from before you started recording, buy it: it is unrecoverable at any price you can pay later.

Everything from today forward is a different decision, and the reason to record it yourself is not cost. It is that a bought tape cannot tell you when it was not listening, and on a delta stream that is the one property everything else depends on.

The shape of the problem

The venue's market channel emits three event types and only the first is self-contained:

A book at any instant is the last book frame plus every price_change since, applied in order. In production, on 1,500 subscribed tokens, that stream runs at about 2,400 frames a second — a measured median inter-frame gap of 1 millisecond.

Which is why a recorder that drops a connection for eight seconds and reconnects quietly is such a dangerous object. It produces a tape from which the reconstructed book is wrong and looks completely fine. No gap, no error, no way for any downstream analysis to know. That is worse than having no data, because no data announces itself.

Note carefully what that hazard is and is not. It is not a claim that the venue drops connections. It is a claim that if anything ever interrupts your listening — the venue, the network, the process, the host — a delta tape will not show it unless you make it show it. Those are different statements, and the difference is the entire point of the next two sections.

Rule one: a gap is a row, not a log line

Every connect, disconnect, resubscribe, error and stall is written into the tape as an event type of its own, alongside the market data and in the same ordering. Not to a logfile — into the record. The row carries which kind of event it was and a cumulative reconnect counter for that connection.

If your tape cannot answer “were you listening at 03:14?”, every number derived from it carries an error bar you cannot compute.

That is defensive engineering, and defensive engineering usually goes unrewarded. This one paid off, in a direction I did not plan for.

What the rule measured: nothing went wrong

I went back through production tape expecting to find a reconnect rate I could quote. Across 4,952,946 frames written on four separate days, by five recorder sessions each subscribing 1,500 tokens over three connections:

Every connection-state row in the sample is a startup. The longest single unbroken stretch inside it is 19 sequential chunk files from one session start — 2,812,047 frames in 19.3 minutes at 2,425 frames a second, with no reconnect row anywhere in it.

What I had half-remembered as reconnects turned out to be groups of three connect rows spaced hours apart: three connections coming up together at each scheduled restart of the recorder. A restart and a reconnect are indistinguishable in a logfile. They are different rows in the tape, and the tape is right.

So the honest finding about the venue is a null: Polymarket's market websocket did not drop a single connection across the frames measured. It is a small sample in wall-clock terms — hours, not weeks — and it says nothing about the venue's trading API, which is a different service with a measured 99.16% uptime and a documented two-hour halt inside the same period. But for the market data feed, over five sessions, the socket held.

That null is the return on rule one. Without gap rows there is no way to separate “the socket never dropped” from “the socket dropped and I never noticed” — and the natural instinct, which is to assume the worse one and write defensive analysis around it, would have been wrong.

A recorder that writes its own absences can be used to falsify your assumptions about the venue. That is a strictly larger capability than recording books, and it is the reason to keep the rule even when every measurement comes back clean.

The incident that was ours, not the venue's

The failure that motivated the rule was real, and it deserves accurate attribution, because it is easy to cite loosely — I have done it myself — as evidence of feed instability. It was nothing of the kind.

A sibling recorder on the same fleet reported healthy for eleven minutes while uploading nothing. The service manager said active. The process journal showed the loop cycling and the fill counter climbing. The object store held nothing at all.

The cause was an IAM policy on our own infrastructure: the instance role granted s3:PutObject on three prefixes and nobody had added the fourth. The process was connected, receiving and parsing correctly the whole time — the uploads were failing. Every signal a person naturally reaches for was true and useless.

That is our bug, not the venue's, and conflating the two is exactly the error gap-recording exists to prevent. Two different things can put a hole in a tape, and they leave opposite evidence:

A recorder that merely logged “disconnected” would have said nothing about either. A recorder that merely counted frames would have reported the venue healthy on a day when eleven minutes of tape was going nowhere.

Two details made that recoverable rather than a lost night, and both generalise: the writer keeps its local copy when an upload fails, and the failure was loud in the process journal even though it was invisible to the service manager. A recorder that deletes on a failure it mistakes for success destroys data while printing healthy cycles.

Rule two: record raw, reconstruct nothing

The temptation on a delta stream is to fold the deltas into snapshots as they arrive and store clean book states. It halves the storage and makes the consumer trivial.

It also bakes today's understanding of the protocol into a tape that cannot be re-derived. When your delta-application logic turns out to be subtly wrong — and it will; a size of “0” meaning removal rather than a zero-size level is exactly the kind of thing discovered late — the folded tape is unrecoverable, and the raw one is just reprocessed.

Both the venue timestamp and the local receive time get stored. They are different clocks and the difference is sometimes the measurement you want.

Rule three: the selector is part of the instrument

A recorder that picks its universe by quotability — spread inside a band, queue under a cap — is selecting for the markets a maker would quote. That is correct for a tape about making, and silently wrong for almost any other question.

Measured on a production universe: 236 temperature tokens across 64 ladders, best coverage five rungs of eleven, and 29 ladders represented by a single rung. No complete ladder had ever been recorded, so no ladder-level question could be asked of that tape at all — and the ladder questions are precisely the ones where the dead rungs carry the information.

On one live ladder, seven of eleven rungs had zero offers against 27–42 bid levels holding four to nine thousand shares. A quotability filter keeps the four rungs where the answer is obvious and discards the seven where it is not.

The fix is a second selection mode that keeps whole events — every rung, dead ones included — with the cap falling on an event boundary, because a half-recorded ladder is worse than none when the sum across rungs is the whole point.

What the venue publishes that nobody subscribes to

The market channel carries eight event types, not three. Four are lifecycle:

Documentation disagrees on how to get them — one source describes them as market-level channels needing no symbol filter, the venue's own docs show no subscription without an asset list and gate them behind a feature flag. So the recorder asks the documented way, writes the exact subscribe payload into the tape, and counts what arrives. A zero is then an answer rather than an assumption.

It arrives. In a 100-second run on 120 subscribed tokens:

Six market creations in 100 seconds — roughly 5,000 a day. And token_id is empty on all six, which settles the protocol question: these frames describe markets the recorder is not subscribed to. A new_market frame carries the condition id, token ids, question, outcomes, tags and tick size — enough to auto-subscribe, which is how a recorder claims to cover “all markets.”

One of the four turns up without being asked for. The 4.9 million production frames above were recorded before the lifecycle subscription existed — no new_market, no best_bid_ask, no market_resolved anywhere in them — and they still contain 36 tick_size_change events. The minimum tick moved under a live book 36 times, and a recorder that discarded event types it did not recognise would have dropped every one. Store what you did not ask for.

Why this matters to a tape of books and prints: such a tape cannot say when a market began or ended, so every study built on it infers both from the first and last frame it happens to contain — a property of when the process was listening, not of the market. Those two instants are what make a tape self-describing.

Cost, for calibration

A production recorder on a few hundred tokens writes about 5.2 GB a day, roughly 1.9 TB a year, uploading about 7 MB every 62 seconds. The venue holds 9,814 tokens at any time. Full coverage is a fan-out and storage problem, not a protocol one — the subscription and the parsing are unchanged.

What this establishes, and what it does not

Does: that the market websocket dropped zero connections across 4.95 million recorded frames on five sessions; that gap-recording is what makes such a statement possible at all, and is therefore load-bearing whichever way the measurement lands; that the only tape-hole actually observed was self-inflicted and invisible to every health signal except a fresh object in the store; that quotability-selected universes cannot answer structural questions; that the lifecycle channels exist and deliver, at about 5,000 market creations a day, with tick_size_change arriving whether you subscribe for it or not.

Does not: that the feed is reliable in general. Five sessions and a few hours of wall-clock is not an uptime figure, and a null result on a small sample is the weakest kind of good news. It is also specific to the market data socket: the same venue's trading API halted for two hours inside the same period, which is a separate service and a separate article.

Nor that self-recording beats buying. For history you do not have, it loses by definition. And every vendor's book coverage starts recently — August 2025 at the earliest, most in 2026 — so nobody has deep prediction-market L2 history.

The honest limitation on self-recording is the one this whole post is about: your tape is only as good as your record of your own absence. Buying moves that uncertainty onto someone else's infrastructure without removing it, and most vendor documentation does not say how they handle it.