How I actually think about systems.
Every system below already shipped. This page is the reasoning underneath it — the constraint that forced the decision, the options that were genuinely on the table, what I gave up to pick one, and the number that proved it worked.
tokens / month through LLM infrastructure
events / month on a self-managed pipeline
keyword SERPs processed monthly
locales served by one collection system
What I optimise for
Unit economics are a design input
At a billion events and five billion tokens a month, the monthly bill is an output of the architecture, not a finance problem discovered later. The two largest wins I have shipped — 62% off AI processing, a five-figure monthly saving on data processing — were both design decisions, not optimisations bolted on afterwards.
Invariants get enforced, not documented
A rule that lives only in a wiki page is a rule that a new engineer will break on their first sprint. When a booking identifier turned out to repeat across accounts, the fix was not a note — it was an explicit account-scoping rule applied to every query path, including raw SQL.
Assume dirty data and duplicate messages
Collected data is gated on row counts, null rates, and duplicate checks before it reaches production, with alerts when a gate fails. Billing webhooks assume every event arrives twice. Neither assumption is pessimism — both are just what happens at volume.
Deterministic beats clever
A clever system that is hard to reason about fails in ways nobody can debug under pressure. A memory-aware job scheduler and brand-level hash partitioning are both deliberately boring choices that make behaviour predictable at scale.
The trade-offs, written down
A decision with no cost attached isn’t a decision — it’s a preference. Each record below names what was given up, not just what was gained.
Move LLM inference off per-token APIs
The constraint
AI Hyper Cube runs scoped prompts over AI-generated search content at 5B+ tokens a month, across 100M+ keyword SERPs. At that volume the dominant cost driver was not compute or storage — it was per-token API pricing, and it grew linearly with every unit of product growth.
Options on the table
Stay on hosted per-token APIs
No serving infrastructure to own — but the cost curve is set by someone else and scales with every token.
Self-host open-source models on GPUschosen
Capacity-based cost and direct control of batching and throughput, at the price of owning model serving.
The decision
Engineered a GPU-based inference platform on vLLM and rented GPU capacity running open-source models for large-scale entity extraction, moving the extraction workload off hosted APIs entirely.
What it cost
Traded a zero-operations dependency for infrastructure I own: GPU capacity planning, model serving, and throughput tuning became my problem — in exchange for a bill that no longer scales with every additional token.
What it moved
- Monthly AI processing cost cut 62%.
- Throughput and extraction accuracy held at 5B+ tokens per month.
Retire BigQuery UDFs for a self-managed engine
The constraint
The legacy BigQuery UDF workflow processed more than a billion events a month. It was expensive at that scale and hard to reason about operationally — and both problems compounded as volume grew.
Options on the table
Keep tuning the BigQuery UDF workflow
No migration risk, but cost and operational behaviour stay inside someone else’s engine.
Self-manage Trino, Iceberg and ClickHousechosen
Direct control of memory, scheduling, and storage layout — at the cost of running the cluster.
The decision
Built a Trino–Iceberg–ClickHouse pipeline on Python, GCS, Hive metadata, JavaScript UDFs and Kubernetes, with a custom memory-aware job scheduler to run distributed jobs safely at that volume.
What it cost
Took on cluster operations and a scheduler I had to write myself, to get predictable memory behaviour and a cost curve I actually control.
What it moved
- A five-figure monthly saving versus the legacy process.
- 1B+ events processed per month on the new pipeline.
Separate scraped data from application data
The constraint
Scraped claims data and application data have different shapes, different lifecycles, and different failure modes. Putting both in one database would have coupled the scrape layer to the product permanently — every pipeline change becoming a product migration.
Options on the table
One database, one schema
Simple joins and migrations; the data layer and the application can never evolve independently.
Separate databases behind an explicit routerchosen
Independent lifecycles — but no cross-database joins, and every query’s destination must be deliberate.
The decision
A dual-database architecture with a custom Django database router: application data and scraped claims data live in separate, independently managed databases.
What it cost
Gave up cross-database joins and added a routing layer every query passes through, in exchange for a data layer fully decoupled from the application layer.
What it moved
- 13 scraped data tables across 2 databases behind one custom router.
- The collection pipeline can change without forcing a product migration.
Treat a non-unique ID as a fact, not a bug
The constraint
A key booking identifier repeats across different accounts in the scraped source data. That is not a defect waiting to be fixed upstream — it is a property of the data. Any query written assuming global uniqueness would silently return another account’s records, and would look correct in every test written by the person who made the assumption.
Options on the table
Assume the identifier is unique
Reads naturally and stays wrong exactly where it matters most.
Scope every query path by account, explicitlychosen
More verbose at every call site — and the invariant cannot be quietly forgotten.
The decision
Designed and documented explicit account-scoping rules for every query path: list queries, detail lookups, ORM subqueries, and raw SQL joins.
What it cost
Every read carries an extra predicate and a rule someone has to know — the price of an invariant that still holds when a new engineer writes the next query.
What it moved
- No account can retrieve another account’s records.
- The scoping model is written down and reviewable, not folklore.
Assume every webhook arrives twice
The constraint
Stripe can and does redeliver the same billing event. A handler that is not idempotent double-applies a plan change; a handler that throws makes Stripe retry work that already succeeded. Both failure modes corrupt billing state, which is the least forgiving state in the product.
Options on the table
Handle each delivery as it arrives
Correct only while the network is — and billing is the wrong place to find out it isn’t.
Key idempotency on the provider’s event IDchosen
Redelivery becomes a no-op; failures must surface through monitoring instead of retries.
The decision
An idempotency model keyed on the Stripe event ID, with handler errors caught and logged rather than re-thrown.
What it cost
Failures surface through logs and alerts instead of provider retries — which means the monitoring has to be real, not aspirational.
What it moved
- The full billing lifecycle — trial, paid, upgrade, downgrade, cancellation, reactivation — runs on idempotent handlers.
- A logic bug never causes Stripe to retry an event that already succeeded.
Things I reach for again
Hash partitioning to avoid hot partitions
Brand-level MD5-modulo partitioning in the Spark post-processing stage keeps analytics queries evenly distributed as brand count grows.
Memory-aware job scheduling
A custom scheduler that admits distributed jobs based on available memory, rather than letting a query engine discover the limit by failing.
Event-driven collection
FastAPI in front, RabbitMQ between, Redis-backed workers under Argo Workflows behind — so collection throughput scales with workers instead of request handlers.
Quality gates before production
Row-count, null-rate, and duplicate checks run after collection and before the data is visible, with Slack alerts on failure. Nothing that fails a gate reaches a customer.
Caching measured by outcome
Varnish in-memory caching then a CloudFront CDN migration, tracked by a single number: good-LCP URLs from 55.73% to 83.85%.
Absorb upstream changes, don’t inherit them
When Google deprecated num=100, an automated AI Overview stitching system absorbed the change and cut collection cost ~60% across 25+ locales.
Where this actually ran
Every decision on this page came out of a system with a problem, a solution, and a measured result. Those are written up in full on the Projects page.