Backend systems
Ledgerline
An event-stream API that keeps writes in order, handles retries without duplicates, and detects changes to saved history.
- Role
- Timeline
- Stack
- Links
By the numbers
99
tests in 15 files, running in Workers with real Durable Objects and a local D1 database
12
documented failure modes, each with its expected behavior and a test
8
bugs found in review, each fixed and covered by a regression test
1
runtime dependency (Hono)
- Client
- Worker
- Durable Object
- D1
- Client to Worker POST /v1
/streams With an Idempotency-Key header./:id /events - Worker Key, rate and owner checks Looks up the API key by its SHA-256 hash, consumes one rate-limit token and checks stream ownership. An empty bucket returns 429 with Retry-After.
- Worker to Durable Object append(payload, key) Routed by idFromName(streamId).
- Durable Object block
Concurrency A reused key with the same body returns the original {seq, hash}. A different body is a conflict. A new key gets seq + 1 and the next hash.While - Durable Object One atomic write Saves the event, key record, minute rollup, stream head and outbox alarm together. Either all are committed or none are.
- Durable Object to Worker {seq, hash}
- Worker to D1 INSERT OR IGNORE Copies the event to D1 immediately so reads usually see a new write. If this fails, the client still receives the append result.
- Worker to Client 201 {seq, hash} A replay returns 200 with Idempotent-Replay: true. A different body returns 422.
- Durable Object to D1 alarm(): outbox sweep After about 5 s, inserts any rows missing from D1. Retries every 30 s while writes fail.
Hash chain
- genesis SHA-256("ledgerline:v2:" + stream
Id) - h1 event 1
- h2 event 2
- h3 event 3
hn = SHA-256(hn-1 | RFC 8785 canonical JSON | n)
/verify rebuilds the chain from genesis and checks every seq, prevHash and hash. Comparing the result with the committed head also catches deleted events at the end.
The idea
Ledgerline is an HTTP API for ledgers, audit trails and other event histories that must stay intact. Events can only be appended. Retried writes take effect once, sequence numbers have no gaps, and a hash chain reveals edits, deletions or a truncated history.
What I built
Each event stream has its own Durable Object on Cloudflare Workers. Inside blockConcurrencyWhile, it checks the idempotency key, assigns a sequence number and links the event into the hash chain. One atomic write saves the event, key record, minute rollup, stream head and outbox alarm. The append either commits in full or writes nothing.
D1 stores a read model for paginated history. The Worker copies each event there immediately. If an insert fails, the copy falls behind, and the outbox alarm keeps retrying until the event is in D1.
Each event’s hash includes its RFC 8785 canonical JSON, the previous hash and its sequence number. The hashing rule is versioned per stream. API keys are stored as SHA-256 hashes. Requests for another key’s stream return 404, and a token bucket per key limits requests with 429 and Retry-After.
The 99 tests run in Workers with real Durable Objects and a local D1 database, including deliberately injected D1 failures. CI runs them on every push to main and every pull request.
The hard parts
-
Retries without duplicate writes
Idempotency keys last as long as their stream. Retrying with the same body, compared in canonical form, returns the original {seq, hash} with Idempotent-Replay: true, however much time has passed. Reusing a key with a different body returns 422 and writes nothing. A concurrency test sends 5 requests with one key and checks that only one event is created.
-
Keeping order across await calls
An await inside a Durable Object, such as a SHA-256 call, can let another append start before the first finishes. I run every append inside blockConcurrencyWhile to preserve order. The test makes the digest yield while 20 appends interleave, then checks for sequence numbers 1 to 20. Removing the guard makes it fail.
-
Updating the hash rule safely
RFC 8785’s own test examples exposed a bug: the original canonicalizer misordered integer-like keys such as "9" and "10". Replacing it would have made /verify fail on untouched streams containing those keys. I shipped chain version 2 instead. Each stream keeps its original version, recorded in the genesis hash, so version 1 streams can still append and verify.
Before and after
| Before | After | Source | |
|---|---|---|---|
| Reusing an Idempotency-Key with a different body | Returned a successful replay of the original event. | 422 idempotency_key_reused, and nothing is written. | Commit ff5270e, October 9, 2026 |
| Delivery to the D1 read model | One insert after the commit; a failed insert could leave a permanent hole. | An outbox alarm, set in the same atomic write, fills missing D1 rows and retries every 30 s until delivery succeeds. | Commits d2404dc and 4650d2a, October 9, 2026 |
| Canonical JSON for the hash | Keys re-sorted, then JSON.stringify, which puts integer-like keys out of RFC 8785 order. | RFC 8785, shipped as chain version 2; streams created under version 1 still verify. | Commits e4db446 and a6923e9, October 9, 2026 |
| Tests in the Workers runtime | 69 in 14 files. | 99 in 15 files, and CI fails if the README or the live page quotes another count. | test/*.test.ts on August 2 and October 9, 2026 |
What’s next
Publish signed checkpoints of each stream head outside the server. /verify could then start from the last checkpoint and detect a consistent rewrite of the entire log.
Try it
Shared approaches
Test against real runtimes and data