Read the contract
Follow the input grain, keys, UTC reporting date and integer AUD cents through the flagship explanation.
Open the data contractData engineering field notes, run on synthetic data
Eight messy rows in. One number you can defend.
Walk the board. Every row is real sample data, and every gate is a decision you can open: the Python, the SQL, the schema, the check and why it was built that way.
Daily order valueA$195.00
8 in = 4 kept
+ 1 duplicate + 3 quarantined
| event | order | amount | decided at | fate |
|---|---|---|---|---|
| e1 | o100 | 100.00 | current state | kept |
| e1 | o100 | 100.00 | collapse repeats | duplicate |
| e2 | o101 | 25.00 | current state | kept |
| e3 | o102 | 70.00 | current state | kept |
| bad1 | o103 | -1.00 | validate | invalid_amount |
| bad2 | o104 | 5.00 | validate | unknown_channel |
| bad3 | NULL | 5.00 | validate | missing_order_id |
| e5 | o105 | 0.00 | current state | kept |
Six field notes on small synthetic order data, so you can check every number by hand.
Field notes
All field notes, and the RSS feedAll sixA daily orders pipeline you can run in DuckDB
Start with a messy delivery and follow each decision to a trusted current order. Inspect validation, replay handling and correction before you run it yourself in your browser.
Topics: Pipeline, Modelling, Quality
6 min read
Updated
An idempotent pipeline counts each order once
When delivery retries, the same event can arrive again or disagree with its earlier payload. Trace how the pipeline recognises a safe repeat, rejects a conflict and protects the committed result.
Topics: Reliability, Pipeline
7 min read
Updated
Late-arriving data can revise an earlier day
A correction can arrive after an order has changed yesterday's total. See how the pipeline chooses the winning state, removes the old contribution and adds the corrected one.
Topics: Modelling, Reliability
7 min read
Updated
Re-run a backfill without doubling the total
Backfill a daily warehouse slice safely: compare append-only SQL with replace-by-slice and watch a replay keep the same total.
Topics: Reliability, Pipeline
7 min read
Updated
Schema drift should fail loudly, not quietly
Schema drift, run in DuckDB: four column changes go through a permissive loader and a data contract, showing which rows vanish, mis-type or stop.
Topics: Quality, Pipeline
7 min read
Updated
dbt snapshots and SCD type 2: what history misses
See a dbt-style snapshot table joined three ways: the current row rewrites history, every version multiplies revenue and an as-of join gets both right.
Topics: Modelling
9 min read
Updated
Four things you can do here
- Open the pipeline diagram
Every stage of the pipeline, one click away.
- Read the code and the reasoning
Why, Python, dbt / SQL, schema and checks on every stage.
- Sketch your own pipeline
Drag blocks, connect them and add notes.
- Run the example on real rows
DuckDB runs it in your browser, and every result shows where it came from.
Trace the claim before you trust the result
Every note keeps the explanation, the code that runs and the results side by side, and says plainly what a small example can and cannot show.
Inspect the SQL
The exact query the notes run, versioned with the sample data and the checks for each stage.
View the bundled SQLKnow the limits
Saved results and your own runs are labelled separately. A small sample in a browser shows how the logic behaves, not how it performs at production scale.
Read the limits