An idempotent pipeline counts each order once

Retrying a delivery is sometimes the safest response to a timeout. If the previous attempt already wrote, the retry arrives with events that may look new but are actually the same business fact. Without replay safety, the same order would contribute twice to a daily total.

This companion story uses the same synthetic retail-orders project as the flagship. The amount is integer AUD cents, dates are UTC, and the orders are not recognised accounting revenue. The pipeline that decides whether a retry is harmless is the one explained here.

Two rows describe the same business event when their event identifier is identical and the full raw payload matches. The pipeline records the raw payload as a SHA-256 hash alongside the event identifier in the event ledger. An empty or missing event identifier is quarantined first; a usable identifier with a different hash is treated as a conflicting payload, not as a duplicate.

• The event identifier alone is not enough: the same identifier with a different payload must be rejected, not silently collapsed.
• The full raw payload is hashed so the comparison does not depend on column ordering, whitespace or JSON formatting.
• Quarantine reasons are mutually exclusive; each invalid row receives one primary reason and stays as its own quarantine record.

The pipeline keeps every original event row long enough to assign one primary quarantine reason to every invalid row, and to compare usable event IDs against the committed event ledger. Only after that does it collapse raw-identical valid events into a single accepted event.

The accepted unique event count is one number; the duplicate count is a separate number; the quarantined count is a third. The summary distinguishes them so the reader can see what the pipeline actually did with the input.

If a batch reuses an event identifier with a different raw payload, either within the same batch or against the committed ledger, the pipeline stops before any mutation. Nothing is written to the event ledger, the order state, the fact table or the daily groups.

This strict rejection is deliberate. Once two competing payloads share an event identifier, the pipeline cannot tell which one was the intended business fact. Continuing would risk accepting the wrong one; rolling back leaves the previous committed state authoritative and gives the operator a clear signal to investigate.

Replaying the same accepted batch against the committed ledger demonstrates idempotent application for this small local example. The unique valid events that were already accepted are recognised; their contribution to the committed state is not added again. The accepted unique event count for the replay is the same as the original batch, and the ignored event count includes those same events.

The current order facts, the daily groups and the quality checks are identical to the values produced by the original batch. The commit point is what makes that possible: the pipeline does not mix partial work into the visible state.

Replay safety is what lets a delivery system retry without auditing the previous attempt. A daily report would otherwise risk double-counting a corrected order, and a dashboard would risk presenting the same order twice. By recognising repeats before mutation, the pipeline can be retried at any point without changing the committed state.

For this synthetic project, replay safety is demonstrated inside one browser session against a small sample dataset. It does not prove exactly-once delivery across a broker, a database, a network or any other production side effect.

The pipeline rejects a conflicting payload for an already-accepted event identifier instead of trusting the most recent source update. The alternative would let the more recent write win, which would simplify the source-code path but would also make the order fact dependent on the order of retries.

Rejecting the conflict preserves an auditable answer: the previous committed state stays visible, the operator is told the event identifier cannot be reused with a different payload, and the rejection is reported in the run summary rather than absorbed into a silent overwrite.

The saved result below is shown only when its build, sample data, schema, SQL and runtime identities match this application. It is a captured browser run from a real pipeline, not a fabricated outcome. Its sample is the follow-up batch, which contains one raw-identical valid repeat and several events that the previous initial batch already accepted.

Replay-safety numbers a reader can read directly off the saved summary: the duplicate count of one is the raw-identical repeat the pipeline collapsed; the three ignored events are the accepted events that were already in the ledger and therefore contributed zero new state. Together they show that the retried batch did not add a second contribution from the same business event.

The grain of the summary is per-batch: input event rows reconcile to invalid rows, identical valid duplicates removed and unique valid events accepted. Current order facts are one row per order; daily groups are aggregates by UTC date and channel. Event totals, fact rows and daily groups are separate grains and should not be added together. The example proves behaviour for these small synthetic inputs and this browser pipeline. It does not establish production scale, accounting treatment, durable recovery or exactly-once delivery. If the engine cannot start, this explanation and the bundled SQL remain available.

Evidence mode: saved browser run captured 2026-10-23T20:36:39.286Z from build 821aed49a48d with sample file SHA-256 464d7430a9cf and SQL SHA-256 cd70dcad203d. New local runs from the flagship are labelled separately.

Read the flagship end-to-end story 

See how late corrections revise earlier days 

Inspect the bundled pipeline SQL 

The architecture fragment below is the part of the flagship pipeline that decides whether a retried batch is a harmless repeat. The event identity gate checks the ledger; validation assigns one reason per invalid row; dedupe collapses only raw-identical valid events; the ledger and current state stages record what the replay contributes.