Market-data feed
- 1Exchange binary feed
- 2Broker socket, on silence threshold
- 3Last trusted book
- 4Refuse to publish
A silent primary is a failure, not a calm market. Past the threshold another source takes the tape.
Each stage opens into what I built there, what it has to survive, and why that matters on a live desk. Below them are the detail figures: service boundaries, order lifecycle, fan-out, the market-data plane, the data platform's validation gate, and latency.
Master, control, OMS, and venue are separate tiers because each failure had to land somewhere specific. Strategy publishes once and never names a broker; control sizes and routes; each OMS owns one venue and every book on it.
← DRAG TO READ THE FULL DRAWING
MASTER
One tradesheet at unit size. No clients, capital, or brokers in the strategy layer — adding any of them is operations, not a fork.
CONTROL
Per client: size, broker, OMS. Routing continues from same-day cache when the directory is down; orders that used the cache are marked.
OMS
One instance per broker, every account on that broker inside it — own connection, cache, rate limit, and lane.
Tick → feature → signal → risk gate → order. Pre-trade risk blocks: no answer, no order. Post-trade risk observes: position, charges, and P&L may lag by a moment — they never hold a trade.
← DRAG TO READ THE FULL SECTION
Hypothesis → backtest with real costs and causality → paper on the live order path → capital. Eight strategies cleared every gate; most ideas stop earlier, and that is the engine working.
← DRAG TO READ THE FULL LOOP
THE ENGINE
Instruments, entries, legs, exits, and lifecycle rules are declared — not forked. Multi-leg structures, Greeks, and strike selection are part of the contract.
THE GATES
Next-bar fills after the broker clock. Charges by product class. Walk-forward freezes training decisions before out-of-sample is scored.
THE HANDOFF
Same OMS, same timeouts, same fan-out. What clears paper is the program that runs on capital.
Most architecture diagrams draw every arrow the same way and hide the only interesting question: which edge is synchronous and on the order path, and which is a stream that can fall behind without anyone losing money. These are typed. The two dashed sockets mark the places the system is designed to grow — a new broker is an adapter, a new strategy is a producer, and neither one edits the order path.
← DRAG TO READ THE FULL MAP
Each figure carries one assertion, and each one is something I would happily be interrogated on.
FIG. A — ORDER LIFECYCLE
The most expensive mistake in a broker integration is treating a timed-out request as a decided one. It is undecided — the order may already be live at the exchange. So the state machine has no edge from TIMEOUT to done. It has an edge to a recheck that runs off the hot path, and only the venue's answer writes the book, with partial fills aggregated correctly.
FIG. B — EXECUTION FAN-OUT
A trader or a strategy fires once. Sizing happens per account, each leg leaves through that client's broker adapter, and isolation is the invariant: one client erroring, or one broker host having a bad morning, must not stall the other lanes. Paper trading leaves by the same door as live, so research and production are the same program.
FIG. C — MARKET-DATA PLANE
A silent primary tape looks exactly like a calm market until you decide it doesn't. Past a silence threshold the switch throws to the standby source and the chain stays priced. Downstream the plane does three separate jobs — sequence, serve now, store for replay — because collapsing them is how research ends up reading a different market than the desk traded.
FIG. I — DATA PLATFORM
Loads land in a dated staging area and are promoted into the system of record only after validation — schema and grain, duplicates, freshness, and gaps diffed against the exchange trading calendar. A run that fails stays in staging and is recorded as degraded rather than half-loading a day that a strategy will later average over. One control centre is the sole trigger authority, because two schedulers firing the same pipeline is a data-corruption bug waiting for a busy morning.
FIG. D — PLANE SEPARATION
Everything a human looks at can be slightly stale. Nothing that places an order waits on it. The crossed edge is the one this drawing exists to forbid: a lookup belonging to the display path quietly becoming a dependency of the send path. When the directory is down, routing continues from same-day cached state and the screen says so.
FIG. E — ORDER-PATH LATENCY
Colocated, the order path sits in the ten-to-twenty millisecond band. Over the internet it is eighty to five hundred. Those are the numbers the desk plans around, and they are the ones worth quoting because they describe the whole path a real order takes rather than one hop of it.
Resilience is not a property you claim, it is an ordered list you can recite. For each subsystem: what is preferred, what takes over, and where it stops. The last rung is always to stop and say so — because a system that carries on with invented state fails silently, and silent is the expensive kind.
← DRAG TO READ THE FULL LADDER
A silent primary is a failure, not a calm market. Past the threshold another source takes the tape.
The order of the tiers is declared, and every result records which tier answered it.
Status degrades in quality rather than disappearing. Undecided stays undecided until the venue answers.
Routing survives the directory being down, and the orders that used the cache are marked as having done so.
A storage outage delays the record. It never rejects the trade.
One broker having a bad morning must not take the other four with it. Lanes are isolated per broker and per client.
History is the one thing that cannot be rebuilt from the market. It is replicated and backed up, not just stored.
Failover, replication and restore get conflated constantly, and the day that matters is the day someone discovers they meant different things. Each tier below runs on its own clock, and each subsystem gets the mechanism its tolerance can actually afford.
← DRAG TO READ THE FULL DRAWING
Failover keeps you trading in seconds and is automatic. Replication means a second copy exists and runs on a schedule you can measure. Restore means trading off that copy, and it is a deliberate act with a human in it. A desk that says “we have a backup” and means the second one has confused durability with availability.
The order path cannot wait for anything, so its continuity is in-process: a silence threshold throws the feed to a standby source, a per-host breaker isolates a failing broker, routing continues from same-day cached state when the directory is unreachable. The warehouse can afford minutes, so its continuity is replication. Matching the mechanism to the tolerance is the whole design.
Primary reads, backup writes, never the reverse under normal operation. A bidirectional copy is a conflict waiting for the day both sides are reachable and disagree. The copy itself is pulled by the backup server rather than streamed through the job that triggers it, so the orchestration process is never the bottleneck or the failure point for the data movement.
Every service addresses its stores through configuration rather than a hardcoded host. Recovering therefore means verifying the replication gap, repointing that configuration at the backup, and restarting — no service is rebuilt, and no service needed to know in advance that it might one day be talking to a different machine.
A green health check proves the host answers, not that its data is caught up. Those are separate checks: continuous deep health probes on both primary and backup, and a separate on-demand replication-gap walk across the estate — on-demand because walking every table and collection is expensive enough that it must not run on a timer and quietly degrade the thing it is measuring.
In a system that keeps gaining services, continuity cannot be per-service heroics. A new service inherits the posture by satisfying three contracts: it declares its dependencies so the topology knows what it takes with it when it fails, it exposes a health endpoint the monitor already knows how to poll, and it addresses its stores through configuration so it is repointable. Meet those three and the service is covered on the day it ships.
A recovery path that has never been executed is an inference from how the configuration works, not a rehearsed procedure — and the honest thing is to say so and schedule the drill, rather than discover the difference during an incident. Replication cadence is the working recovery-point target until a tighter one is defined and tested.
“Robust” is not a property. It is a set of specific things that have gone wrong and now have a test with their name on it. These five shape the architecture.
Gap, duplicate and reorder are named failures on the feed, not surprises. A book built on a missing packet is not a book.
A silent primary is a failure, not a calm market. Past the threshold the standby takes the tape and the chain stays alive.
Mark it done and the fill that lands later never reaches the book. Recheck the venue off the hot path, then write the state.
Repeated failure trips the lane open per host, so one broker having a bad morning cannot take the other four down with it.
The order commits locally first and replicates behind. A storage outage delays the record; it does not refuse the trade.
ASK ME ABOUT
how sizing resolves per client; what the order path does when the client directory hangs; why a timeout is undecided rather than failed; how an order survives a storage outage; why a modify never quotes off last price; what is empty at 09:14 if an overnight job died.
Anyone can say they avoid look-ahead. The question worth asking is what happens when somebody forgets. These are the places that answer is “the build fails” or “a test goes red” rather than “someone notices in review”.
← DRAG TO READ THE FULL DRAWING
The strategy core cannot read data later than the current timestamp, and it cannot import an adapter — an import-contract check fails the build if it tries. The same constraint that prevents look-ahead is what makes live an adapter swap rather than a rewrite. It is one rule wearing two hats.
A missing bar stays missing. Rows are wholly present or wholly absent, and a lookup returns nothing rather than the last value carried forward. A fabricated price can fire a stop that never happened, or hide one that did — and both of those show up as performance rather than as an error.
Walk-forward freezes every decision made inside a training window before test sessions are scored, so a parameter chosen because it worked cannot then be credited for working. Anything that only survives when that freeze is relaxed did not survive.
An index is not a fixed list. Members join and leave, and a backtest whose universe is built from today's constituents has quietly tested a portfolio nobody could have held. So constituent membership and weights are stored historically and read as of the trade date, and expired or delisted contracts stay in the store rather than being cleaned out of it. The instruments that disappeared are the ones carrying the losses.
Lot sizes, tick conventions and instrument masters change, and the danger is not that a change breaks something loudly — it is the day the engine reads the new lot size from the chain while a report divides by the old one from config. Both numbers are internally consistent, the pairing is false, and nothing surfaces it. Reference data therefore carries its own effective dates and the values asserted in code are catalogued against what reads them.
Run enough permutations across strikes, times, thresholds and symbols and something will look excellent by construction. A scan that reports only its winner is reporting a maximum, not an edge. So a sweep records how many configurations were evaluated alongside the one that won, and the winning row carries the hash of the configuration that produced it — the count of trials is evidence, not a footnote.
Choosing a parameter, a symbol or a strike-selection method is itself a decision that had to be made at a point in time. Walk-forward freezes those decisions inside the training window before test sessions are scored, and the same rule applies to the universe: whatever narrowed it had to be knowable then. A filter applied with hindsight is indistinguishable from a good result.
A contract is identified by underlying, expiry, strike and type — everywhere, including deduplication, the quality gate and the cache. A partial key quietly merges two contracts, because the same strike exists across every expiry listed on a trade date, and the resulting series looks perfectly plausible.
Every run stamps its resolved configuration into a hash, and live refuses to start unless that hash matches an explicitly frozen one. It turns “did we deploy what we tested?” from a memory question into a startup assertion, and kills the failure where someone edits a live parameter weeks later and the backtest evidence quietly stops applying.
Values the exchange owns — session bounds, tick conventions, lot sizes — are held apart from anything a strategy owner may tune, and the ones asserted in code rather than read from the instrument master are catalogued with what reads them and how they fail. The dangerous change is not the one that breaks loudly; it is the one where every number stays internally consistent and only the relationship to the market is false.
Backtest and live cannot be identical everywhere — one has bar highs, the other has ticks; one models charges, the other queries them. Those places are enumerated, the charge model's correctness is defined as agreement with the live API and enforced by a test, and where selection can legitimately differ, paper runs beside a same-day backtest with the difference diffed rather than assumed away.
Research and simulation engine
CLI + trader-facing builder → contract → warehouse or CSV → features → strategy legs → lifecycle engine (bar-by-bar; portfolio / strategy / group state) → fills (default next bar) → Indian cost factories → metrics. Drawdown kill blocks new entries. Paper is not a second program.
contract / UI → load bars + chain → generate_signals (legs, not prices) → lifecycle (exits / hedge / re-entry / EOD) → fill after broker-ack clock → costs by product class → paper OMS → live OMS
Warehouse is source of truth: a bounded session is sliced, quality-gated, then a dense in-memory cube. Features are timestamp-safe. An event-driven simulator walks the session. Walk-forward freezes train-window decisions before they touch test sessions. Costs last. Then paper, then the five-broker door. No clocks or PnL on this page.
warehouse slice → quality gates → dense cube → timestamp-safe features → event-driven session sim → walk-forward (train frozen) → costs → paper / live OMS
Time is first-class state: the calendar knows it is expiry; the engine will not carry a book overnight; flatten is an EOD contract. Slots and re-entry live in the same lifecycle engine. Same costs. Same kill-switchable OMS. Clock-driven, flatten-or-die — not cube-shaped. Parameters stay at the desk.
Calendar: expiry day?
→ session clock as state
→ same lifecycle engine
time exits + EOD flatten
no overnight positions
→ costs → paper → live OMSGate A
A reason the market might pay you. Not a backtest yet.
Gate B
Next-bar fill. Indian costs. Walk-forward. Fail here on purpose.
Gate C
Same OMS door as live. Simulated fills. Same timeouts, same fan-out.
Gate D
Capital. Eight made it. The rest stay off the book.
Each layer gets the tool that suits its job: C++ where the packet path is hot, Python across the services, a column store for time-series, a cache for what has to be instant.