The data platform and its control centre
Behind the tape is a platform: batch pipelines that load history and reference data, always-on distributors that hold a socket open, a control centre that is the only thing allowed to trigger either, and quality machinery that knows when a day is missing before a strategy is fitted to it.
- ticks / day stored
- 300–600M
- pipelines orchestrated
- ~19
Built here.
- 01
Around nineteen batch pipelines registered in one place — spot, futures and options loads, Greeks and implied-volatility enrichment, corporate actions, index constituents, instrument masters, exchange regulatory reports and retention cleanup.
- 02
Always-on live distributors alongside the batch jobs, including a C++ service that joins the exchange multicast directly and builds a stateful option chain with its own pricing, and a Python service that streams full index chains from a broker socket.
- 03
A control centre that is the sole trigger authority: the batch executor's own scheduler is deliberately disabled so two schedulers can never fire the same pipeline concurrently. One place starts things, one place records what happened.
- 04
A staging-then-promotion path into the warehouse — data lands in a dated staging area, is validated, and is only then promoted into the tables strategies read. Nothing reaches the system of record unchecked.
- 05
Quality machinery with history behind it: pipeline event and alert tables, per-pipeline freshness reporting, and a gap detector that diffs what actually exists against the exchange trading calendar and classifies each missing date by whether it can still be filled.
- 06
Client and desk reporting: broker trade and settlement reports pulled per broker into a common shape, plus exchange reports, corporate-action factors and charge reconciliation — so a client statement reconciles rather than approximately agrees.
- 07
Operational scaffolding treated as part of the product: container healthchecks, restart policies, drain time on shutdown so in-flight queues finish, backup replication for both stores, a written disaster-recovery path and alerting to where the desk actually reads it.
- 08
A documented domain language for the platform — what a settled generation is, what a grain is, which tier served a read — so a disputed number can be traced instead of argued about.
Failure modes.
- A source publishing late, twice, or in a changed shape, which is recorded as a degraded ingest run rather than silently loading a partial day.
- A missing trading day, which the gap detector surfaces against the calendar instead of leaving a hole for a backtest to average over.
- Duplicate rows and renamed columns from upstream, which have named correction paths rather than manual patching.
- A crashed always-on producer, which the restart policy brings back — nothing waits for a human to notice.
New writes default to dry-run: the real session, the real socket and the real computation all run, and only the terminal write is withheld and counted. Gate the writes, never the reads. A fresh deployment cannot touch real data until someone decides it should.