DataEngr
Subscribe

How it works

Four things you can do here.

DataEngr shows how data pipelines are designed, tested and proved. Open the diagram and follow a pipeline stage by stage, read the code and the reasoning behind each choice, and run the pipeline yourself.

  1. 01

    Open the pipeline diagram

    Every field note has a diagram of the pipeline it explains. Point at a stage to preview it, or click it to keep it open. On a phone the stages become a list.

    Open the retail-orders architecture 
  2. 02

    Read the code and the why

    Each stage carries four tabs: the reasoning (the decision, the alternative and the consequence), a Python reference, the dbt model or the SQL that actually runs and the schema. Three of the eight stages, the ones that run a quality check, add the checks.

    See the tabs on a stage 
  3. 03

    Sketch your own on the canvas

    Drag blocks from the same catalogue onto a board, connect them, and write notes in Markdown. Connections that would not make sense in a real pipeline are refused with a reason.

    Open the canvas 
  4. 04

    Run the example on real rows

    The flagship note runs its pipeline in your browser with DuckDB-Wasm on a small sample dataset. You get the counts, the orders, the daily totals and the quality checks from that run, and exactly which build and data produced them.

    Run the pipeline 

What runs and what is illustration

Runs in your browser
Only the SQL marked Executed in your browser. It runs on sample data that ships with the site, with limits on rows, size and time, and every result shows which build and data produced it.
Illustrative
Python references, dbt models, the pipeline diagrams, playback animations, the home page walkthrough and anything on the canvas. They explain a mechanism; they never execute.
Synthetic
The retail-orders data is a small, deliberately messy teaching dataset. Its numbers describe that sample, not any real company or production system.

Start here

  1. A daily orders pipeline you can run in DuckDB The flagship: the whole pipeline, explorable and runnable.
  2. An idempotent pipeline counts each order once Why a retry is safe here and what makes two events the same.
  3. Late-arriving data can revise an earlier day How a correction moves an order between days without double counting.