Data engineering patterns, each with the notes that prove it
A pattern hub explains one concept in plain words, shows a small worked example and links to every field note and tool for it.
Pattern hubs
- Dimensional modelling
What a star schema is, how a snowflake schema differs and why the grain of a fact table matters, shown on the retail-orders star at two grains.
- Slowly changing dimensions
Slowly changing dimension types 0 to 3 in plain words, how dbt snapshots build SCD type 2 and the joins that quietly rewrite the history it keeps.
Every pattern
Each pattern with published material, with its notes, its tools and its hub.
| Pattern | Field notes | Tools | Hub |
|---|---|---|---|
| Slowly changing dimensions |
| Slowly changing dimensions | |
| Backfills | |||
| Schema drift | |||
| Deduplication |
| ||
| Join fan-out | Dimensional modelling | ||
| Late data |
|
Slowly changing dimensions
- Field notes
- dbt snapshots and SCD2
- Tools
- YAML to JSON, which checks the snapshots entries of a dbt properties file
Backfills
- Field notes
- Backfills without duplication
Schema drift
- Field notes
- Schema drift contracts
Deduplication
- Field notes
- Idempotency and replay; Daily report from messy events
- Tools
- Pipeline studio, which drops a duplicated event in a real run
Join fan-out
- Field notes
- dbt snapshots and SCD2
Late data
- Tools
- Pipeline studio, which applies a correction to an earlier day