A timeout is not a fill

The naive version treats the timeout as a terminal state. The request failed, therefore the order failed, therefore mark it rejected and move on. It is a single branch and it reads perfectly reasonably in a code review.

Then one morning the fill lands. The exchange matched your order, the broker booked it, and your system has already decided that order does not exist. Now you are carrying a position nobody is watching. Your risk view is wrong, your exposure is wrong, and the strategy that placed it thinks it is flat. If it is a strategy that re-enters on being flat, it enters again — and now you are carrying two.

The inverse is just as bad. Treat the timeout as filled, and you book a position that was never opened. Every subsequent decision — sizing, hedging, the square-off at close — is computed against a book that does not match the exchange.

Both failures come from the same mistake: collapsing three states into two.

The states an order can be in, from your side of the socket, are:

Unknown is a real state with its own behaviour, not a temporary embarrassment on the way to one of the other two. It needs to exist in the schema, be visible on the blotter, and be something the system will act on by itself.

The rule that follows is short: the venue decides. Not the clock, not the client library, not the operator. Your job when you hit unknown is not to guess — it is to go and ask, and to keep asking until you get an answer you can write down.

The obvious implementation is to retry inline: catch the timeout, query the order book, resolve the state, return. It is also wrong, and for a reason worth being precise about.

The send path has one job and a budget measured in milliseconds. If a broker is slow enough to time out one request, it is very likely slow enough to time out the next one, and the query you are about to make goes to that same broker. You have taken a path that was supposed to be fast and made its worst case a function of the failure you were trying to handle. Under load — which is exactly when this happens — inline recovery turns one slow order into a queue of them.

So the resolution runs somewhere else. The order is marked unknown, the send path returns, and a separate reconciliation loop owns the problem from there. It rechecks on its own schedule, it is allowed to be slow, and nothing waiting to trade is waiting on it.

This is the same principle that separates the control plane from the order path everywhere else in the system. Anything a human reads can be a moment behind. Anything that places an order cannot wait on anything that might be.

To recheck an order you need to identify it, and the identifier the broker uses is generally the one it returns in the response you never received. So the first practical problem is finding an order you cannot name.

That is what client-side identifiers are for: a tag you generate before sending and attach to the request, so a scan of the order book can find your order by something you chose rather than something you were told. Where the broker does not support that, you fall back to matching on the tuple you do control — instrument, side, quantity, price, and a time window — which is weaker, and gets weaker still if the same strategy sends similar orders in quick succession.

The second problem is that the answer arrives in different shapes depending on where you ask. A live status socket is fastest but can drop. A cached snapshot is cheap but stale. The order book is authoritative but heavier. A direct status query is the most precise and the most rate-limited.

So status resolution is layered rather than singular: try the fastest source, fall through to the next when it cannot answer, and record which tier answered. That last part matters more than it looks — when someone disputes a number three days later, the useful question is not what the system said but which source it said it from.

Nothing about this gets simpler when the answer comes back. Orders sit at partially filled far more often than the clean examples suggest, and on Indian venues a single order can be split by the broker into child orders against the exchange freeze quantity — so what you sent as one order comes back as several, each with its own state and its own filled quantity.

Resolving the parent means aggregating the children: summing filled quantity, computing a weighted average price across them, and deciding the parent's state from the set rather than from any one member. A parent whose children are two filled and one still working is not filled, and it is not rejected either. It is still unknown, and the loop is not done with it.

This is also where double-counting creeps in. A recheck that runs while a late status message is arriving can book the same fill twice. The write has to be idempotent on the venue's own identifiers, so replaying the same answer produces the same book.

One timeout is a hiccup. A sequence of them is a statement about the broker, and continuing to hammer a host that is failing makes its recovery slower while consuming the capacity your other clients need.

So failures are counted per host, and past a threshold that lane opens: requests stop going out for a cooling period and fail fast instead. Isolation is the point — the lanes are per broker and per client precisely so that one broker having a bad morning cannot take the other four down with it. A desk with five broker integrations should be able to lose one and keep trading.

The lane closes again on its own, but the open should be loud. Something a human reads has to say that a broker is out, because the strategies routed through it are now effectively paused and nobody should discover that from a P&L number at close.

Everything above is best-effort. The backstop is a periodic reconciliation of your book against the broker's, because the broker's book is the source of truth and yours is a cache of it.

That comparison is the honest test of the whole design. If a position appears at the venue that your system does not have, something upstream mishandled an unknown. If your system holds a position the venue does not, the same. Either way the reconciliation is what catches it, and the fact that it runs at all is the reason the rest of the machinery is allowed to be imperfect.

It is also the diagnostic that matters after an incident. Not the log line, not the alert — the answer to whether the blotter matched the exchange.

This is a lot of machinery for a case that fires rarely. An unknown state in the schema, a reconciliation loop, client-side identifiers, layered status resolution, child-order aggregation, idempotent writes, per-host breakers, and a periodic comparison against the venue. All of it exists for a branch that is quiet on most days.

The argument for paying it is that the failure is silent, asymmetric and correlated. Silent, because the wrong version does not throw — it just quietly disagrees with the exchange. Asymmetric, because a mishandled fill is an unhedged position rather than a missed opportunity. Correlated, because timeouts cluster exactly when the market is busy, which is exactly when your positions are largest.

It is also the thing that separates a system running client capital from a system that works in a demo. Handling the happy path is table stakes. What a desk pays for is the behaviour in the ten minutes a year when the broker stops answering and the market does not stop moving.

You cannot wait for a real timeout to find out. The failure has to be something you can produce on demand, so the broker layer sits behind an interface a test can stand in for: a stub that accepts an order and then simply never answers, one that answers after the order is already resolved, one that returns partial children, one that returns the same fill twice.

The tests worth writing are named after the failure, not the function. Whether a timed-out order is ever marked terminal without the venue being asked. Whether a late fill still reaches the book. Whether replaying a status message twice changes the position. Whether a host that fails repeatedly stops receiving traffic, and whether the other lanes keep working while it does.

A test suite that reads like a list of the ways this has gone wrong is worth more than one that reads like a list of the methods on the class.