Cursors, sequences, and the delivery guarantee we actually offer

"Exactly-once delivery" is the feature everyone asks for and nobody can give you across a network. Here is the guarantee Runnev actually offers, at-least-once with idempotent writes, why that is the honest maximum, and how cursors and sequences make it as good as exactly-once in practice.

The honest answer

Delivery over a network that can drop, reorder, and duplicate packets, to a consumer that can crash between receiving a message and recording that it did, cannot be exactly-once at the transport layer. Anyone who tells you otherwise is either doing dedup somewhere you are not looking or is about to lose your data at the worst moment. What you can build is at-least-once delivery plus idempotency, and then the visible behaviour is indistinguishable from exactly-once for any consumer that dedups on a key.

Sequences: the key you already have

Every batch in a stream has a sequence number, assigned when it is accepted and never reused. That number is the dedup key. A consumer that remembers the highest sequence it has fully processed can reject anything at or below that number as a duplicate, in one comparison, with no coordination and no server round trip.

the entire dedup logic
last = load_cursor()          # highest seq fully processed, -1 if none
for batch in runnev.subscribe(stream_id, cursor=last):
    if batch.seq <= last:      # duplicate from a reconnect
        continue
    handle(batch)              # your side effect
    last = batch.seq
    save_cursor(last)          # commit AFTER the side effect

The ordering of the last two lines is the whole trick, and it is worth staring at. You commit the cursor after the side effect. If you crash between handle and save_cursor, you replay that batch on restart, which is safe because it is at-least-once and your handle tolerates a repeat. If you committed the cursor first and then crashed, you would skip the batch, which is at-most-once, and you would lose data. At-least-once is a choice, and it is the safe one.

Idempotency on the write side too

The same idea protects the producer. Publishes are keyed on (stream, seq), so a producer that derives the sequence from its own monotonic source, a database row version, a log offset, an incrementing counter, can retry a publish after an uncertain timeout and know the result is one batch, not two. The duplicate is detected by the server and reported as "duplicate": true.

This closes the loop. The producer cannot accidentally write a batch twice, and the consumer cannot accidentally process one twice, and neither side needed a distributed transaction to get there. Both properties fall out of the same design decision: make the sequence number meaningful and let both ends key on it.

What we refuse to claim

We will not put "exactly-once" on the marketing page, because for a consumer that does no dedup it is not true, and a guarantee that requires a footnote to be true is not a guarantee. What we will say is precise: delivery is at-least-once, writes are idempotent on (stream, seq), and a consumer that dedups on the sequence, committing its cursor after the side effect, gets exactly-once processing. That is the property you actually wanted, stated in a way that survives a crash.

When you see a gap

One consequence to be aware of: because retention evicts old batches, a consumer that falls far enough behind can resume from a cursor older than the oldest surviving batch. It will then receive the oldest survivor next, and the sequence will jump. That jump is not a lost message you can recover; it is a signal that your consumer was down longer than your retention window. Treat a sequence gap as an alert worth paging on, and size retention for your worst realistic outage, not your average one.

← All posts