Reliable Message Delivery in NATS JetStream: Acks, Retries, Dead Letters, and Replay

JetStream builds delivery guarantees from a small set of primitives that you compose yourself. The happy path proves very little about them; what matters is what happens when a message fails. This post walks the complete lifecycle of a failing message in NATS JetStream, from the publish acknowledgment through redelivery, a dead-letter queue (DLQ), and replay, with working Go.

TL;DR: JetStream’s delivery guarantees compose from a handful of primitives. A publish acknowledgment confirms an event is persisted in a stream (with Nats-Msg-Id deduplication inside a configurable window). A durable consumer tracks each subscriber’s progress, and explicit acknowledgment gives the consumer three verbs: ack (done), nak (redeliver immediately, or after a delay you specify), and term (stop trying); a consumer-level BackOff schedule spaces redeliveries when an ack never arrives. MaxDeliver bounds redelivery. When a message exceeds it, the server publishes an advisory on $JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.<stream>.<consumer>; capture those advisories in a stream of their own, fetch the original message by sequence, and you have a dead-letter queue that is itself a replayable stream. Any authorized consumer can replay history from a sequence number or a timestamp. NATS has no one-line DLQ config; this post builds one in about thirty lines of Go.

NATS does not ship a single-config dead-letter queue. It ships the pieces you assemble one from, and the result is a replayable stream.

The failure scenarios

Delivery goes wrong in a handful of ways, and each has a primitive that answers it.

Code is Go, using the jetstream package; the same primitives exist in the other NATS client libraries.

The lifecycle of a failing message

Before the mechanics, here is the map. A message that fails travels a defined path, and every arrow in it is one of the primitives above.

1
flowchart LR
2
  P[Publisher] -->|publish| S[(Stream)]
3
  S -->|PubAck: persisted| P
4
  S -->|deliver| C[Consumer]
5
  C -->|ack| S
6
  C -->|nak / nak with delay| S
7
  C -->|term| S
8
  S -->|MaxDeliver exceeded| A[[MAX_DELIVERIES advisory]]
9
  A -->|captured by| AS[(Advisory stream)]
10
  AS --> H[DLQ handler]
11
  H -->|get by seq| S
12
  H -->|republish| D[(DLQ stream)]
13
  D -->|replay / repair| P

A message is persisted, delivered, and acknowledged. When the acknowledgment never comes, it is redelivered, then bounded, then routed to a durable place you can inspect and replay. Each step is a configuration value or a method call.

How do acknowledgments work in NATS JetStream?

JetStream has two acknowledgments, and they are easy to conflate: the publish acknowledgment protects the write, and the consumer acknowledgment protects the read.

The publish acknowledgment (PubAck) is the server’s confirmation that your event has been persisted to the stream according to that stream’s storage and replication configuration. On a replicated stream, the PubAck means the message is committed by a quorum (majority) of the stream’s replicas, not merely accepted by one node. If you do not receive a PubAck, you do not know the write landed, so you retry.

Retrying safely is where deduplication comes in. Set the Nats-Msg-Id header on publish, and the stream rejects a duplicate with the same ID inside its deduplication window (configurable per stream, a couple of minutes by default). The PubAck for a duplicate comes back flagged as such, so a retry after a dropped ack is idempotent at the stream boundary.

1
ack, err := js.Publish(ctx, "orders.created", data,
2
    jetstream.WithMsgID("order-8f2c-created"))
3
if err != nil {
4
    // No PubAck: safe to retry, dedup window protects against a double-write.
5
    return err
6
}
7
if ack.Duplicate {
8
    // This exact message ID was already persisted within the window.
9
}

This is exactly-once publish within the deduplication window. It does not make end-to-end processing exactly-once; no distributed system can promise that. What you build on top is effectively-once processing: at-least-once delivery combined with idempotent consumers.

The consumer acknowledgment is the read-side counterpart. With an explicit acknowledgment policy (AckExplicitPolicy), the consumer must positively acknowledge each message, and until it does, JetStream considers that message in-flight and owes it redelivery. The consumer’s durable cursor advances only past messages that are acknowledged, terminated, or bounded out by MaxDeliver, which is why a subscriber that was offline at publish time resumes from its last acknowledged position rather than losing anything.

Two settings govern the in-flight window:

• AckWait: how long the server waits for an acknowledgment before it assumes the delivery failed and redelivers.
• MaxAckPending: the ceiling on unacknowledged messages a consumer may hold at once. This is your backpressure knob; it bounds how much work a slow or wedged consumer can pull without confirming.

The three consumer verbs

Explicit acknowledgment gives the consumer three ways to respond to a message.

The distinction between Nak and Term is judgment, and it belongs in your handler. A timeout talking to a database is a Nak: try again shortly. A message that fails schema validation is a Term, because no number of retries fixes malformed input.

1
msg, err := consumer.Next()
2
// ...
3
switch result := process(msg); {
4
case result == nil:
5
    msg.Ack()
6
case errors.Is(result, errTransient):
7
    msg.NakWithDelay(5 * time.Second)
8
case errors.Is(result, errPermanent):
9
    msg.Term()
10
}

How do I retry a failed message with a backoff?

A fixed AckWait gives you uniform redelivery timing, but transient outages rarely want uniform timing. You want to back off as failures persist. JetStream supports this at the consumer level with BackOff, a slice of durations on the consumer configuration. The first redelivery waits BackOff[0], the second waits BackOff[1], and so on; once the schedule is exhausted, the last interval repeats until the delivery bound is reached. BackOff applies only when the ack wait expires; a plain Nak skips it and redelivers immediately, and NakWithDelay uses the delay you pass instead of the schedule. To back off on a known transient error, either don’t ack (let BackOff drive the retry) or compute the delay yourself from the message’s delivery count and call NakWithDelay.

1
cfg := jetstream.ConsumerConfig{
2
    Durable:       "order-processor",
3
    AckPolicy:     jetstream.AckExplicitPolicy,
4
    MaxAckPending: 1000,
5
    MaxDeliver:    5,
6
    BackOff: []time.Duration{
7
        1 * time.Second,
8
        5 * time.Second,
9
        30 * time.Second,
10
        2 * time.Minute,
11
    },
12
}

When BackOff is set, it replaces AckWait: the server rewrites AckWait to BackOff[0]. If the create request sets the pedantic flag, the server instead rejects the consumer unless AckWait equals BackOff[0]. An omitted AckWait defaults to 30 seconds before that check runs, so in pedantic mode set it explicitly to match. Each entry is the ack deadline for that delivery attempt, starting with the first. Here, a message that isn’t acked within 1 second of its first delivery is redelivered, even if your handler is still working on it. Make BackOff[0] longer than your worst-case processing time, or call msg.InProgress() to extend the deadline on long-running work.

When you set BackOff, size MaxDeliver deliberately against the length of the schedule. A MaxDeliver of 5 with a four-entry backoff gives four spaced retries after the initial delivery; the fifth attempt is the last before the message is bounded out. Size the schedule to the outage you want to survive. With the config above, the last attempt happens about two and a half minutes after the first delivery. About two minutes after that, when that attempt’s ack deadline (the repeated 2-minute tail) expires, the message is bounded out: the server publishes the max-deliveries advisory and the consumer’s ack floor moves past it. To ride out an hours-long outage, extend the BackOff tail and MaxDeliver so their combined span covers the outage. You can also leave MaxDeliver unbounded, or pause the consumer while the dependency is down. Otherwise, messages that run out of deliveries during the outage do not wait at the cursor. They are recoverable only through the DLQ path below.

How do I stop a poison message from redelivering forever?

Some messages never succeed no matter how patiently you retry. That is the classic poison message, and without a bound it redelivers forever and the consumer makes no forward progress. MaxDeliver is the bound: after that many delivery attempts, JetStream stops redelivering the message on its own.

A message that silently falls off the end of its redelivery budget is a message you have lost track of. When a message exceeds MaxDeliver, the server publishes an advisory:

1
$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.<stream>.<consumer>

Related advisories fire on explicit termination ($JS.EVENT.ADVISORY.CONSUMER.MSG_TERMINATED.<stream>.<consumer>) and on negative acknowledgment ($JS.EVENT.ADVISORY.CONSUMER.MSG_NAKED.<stream>.<consumer>). Advisories are transient messages. If nobody is subscribed when one fires, it is gone, so persist them into a stream.

Does NATS have a dead-letter queue?

No. There is no dead_letter: true field on a consumer. You assemble a DLQ from primitives, and what you build is an ordinary JetStream stream with the retention, filtering, and replay that implies.

The pattern is three parts:

1. Capture the advisories. Create a stream that subscribes to the max-deliveries advisory subject. This stream is the index of everything that exhausted its redelivery budget.
2. Fetch the original message. Each advisory carries the source stream name and the stream sequence of the failed message. Get the message by sequence (GetMsg) to pull the original payload back out.
3. Republish to a DLQ subject captured by its own stream, so the dead-lettered messages live in a durable, queryable place.

First the two streams, a few lines of configuration:

1
// Index stream: persists max-deliveries advisories.
2
js.CreateStream(ctx, jetstream.StreamConfig{
3
    Name:     "DLQ_ADVISORIES",
4
    Subjects: []string{"$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>"},
5
})
6

7
// Dead-letter stream: the durable home for failed payloads.
8
js.CreateStream(ctx, jetstream.StreamConfig{
9
    Name:     "DLQ",
10
    Subjects: []string{"dlq.>"},
11
})

Now the handler that ties them together:

1
type maxDeliverAdvisory struct {
2
    Stream    string `json:"stream"`
3
    Consumer  string `json:"consumer"`
4
    StreamSeq uint64 `json:"stream_seq"`
5
}
6

7
// advisories is a durable consumer on the DLQ_ADVISORIES stream.
8
func runDLQ(ctx context.Context, js jetstream.JetStream, advisories jetstream.Consumer) error {
9
    _, err := advisories.Consume(func(m jetstream.Msg) {
10
        var a maxDeliverAdvisory
11
        if err := json.Unmarshal(m.Data(), &a); err != nil {
12
            m.Term() // malformed advisory: never retry
13
            return
14
        }
15
        src, err := js.Stream(ctx, a.Stream)
16
        if err != nil {
17
            m.Nak()
18
            return
19
        }
20
        orig, err := src.GetMsg(ctx, a.StreamSeq) // get by sequence
21
        if err != nil {
22
            m.Nak()
23
            return
24
        }
25
        dead := nats.NewMsg(fmt.Sprintf("dlq.%s.%s", a.Stream, a.Consumer))
26
        dead.Data = orig.Data
27
        dead.Header = nats.Header{}
28
        for k, v := range orig.Header {
29
            // Drop publish-control headers: dedup ID, Nats-Expected-*, TTL, rollup, etc.
30
            if k == jetstream.MsgIDHeader || strings.HasPrefix(k, "Nats-Expected-") ||
31
                k == "Nats-TTL" || k == "Nats-Rollup" {
32
                continue
33
            }
34
            dead.Header[k] = v
35
        }
36
        dead.Header.Set("Nats-Dlq-Origin-Seq", strconv.FormatUint(a.StreamSeq, 10))
37
        // Unique per stream/consumer/seq: two consumers dead-lettering the same message
38
        // both land, and a redelivered advisory is still deduplicated.
39
        dead.Header.Set(jetstream.MsgIDHeader, fmt.Sprintf("%s.%s.%d", a.Stream, a.Consumer, a.StreamSeq))
40
        if _, err := js.PublishMsg(ctx, dead); err != nil {
41
            m.Nak()
42
            return
43
        }
44
        m.Ack()
45
    })
46
    return err
47
}

That is the whole DLQ. The handler is itself a well-behaved JetStream consumer: if it cannot reach the source stream, it Naks, which redelivers the advisory immediately (use NakWithDelay to space those retries); if the advisory is malformed, it Terms. Failed messages land on dlq.<stream>.<consumer>, subject-partitioned so you can inspect or replay a single failing consumer’s backlog without touching the rest.

Because the DLQ is a stream, you work with it through another consumer: read the dead-letter stream, fix the downstream defect, and replay the affected messages back onto their original subject.

Retention interactions

The DLQ pattern touches the source stream by sequence, so how that stream retains messages matters.

• Limits retention: the stream keeps messages until an age, size, or count limit is hit, independent of acknowledgment. For the DLQ handler, when the advisory fires, the original message is still there to fetch, as long as your limits are generous enough that it has not aged out between the last delivery attempt and the handler’s get by sequence.
• Work-queue retention: a message is removed when it is acknowledged or terminated (Term). Running out of MaxDeliver does neither. The server stops redelivering and moves the consumer’s ack floor past the message, but the message stays in the stream. When the advisory fires, the handler’s get by sequence will find the original. The original then stays until a stream limit (MaxAge, MaxMsgs, MaxBytes) evicts it or something deletes it, and in the meantime it counts against those limits. Once the copy is safely published to the DLQ stream, add a step to the handler that deletes the original from the source stream (stream.DeleteMsg(ctx, seq)).
• Interest retention: a message is retained until every consumer interested in it has acknowledged it. A consumer that bounds a message out with MaxDeliver never acknowledges it. The original therefore stays available for the handler’s get by sequence, even after other consumers ack it. The cost is the same as above: dead-lettered messages pile up until limits or a delete remove them. Deleting the original here removes it for every consumer, so only do that once you know the other consumers are done with it, for example when the DLQ’d consumer is the only one filtering that subject. Otherwise, leave cleanup to the stream’s MaxAge.

The general rule: the original stays in the source stream only until a stream limit evicts it or something deletes it, so capture the payload while it still exists. Copying the original into the DLQ stream at capture time, as the handler above does, is what makes the dead-lettered data durable regardless of what the source stream does next.

How do I replay past messages in JetStream?

Replay is the same read primitive, aimed backward in time. A consumer’s DeliverPolicy sets where it starts:

• All: from the first message in the stream.
• Last: from the most recent message.
• New: only messages arriving after the consumer is created.
• By start sequence: from a specific stream sequence.
• By start time: from a specific timestamp.

ReplayPolicy then controls pacing: instant delivers as fast as the consumer can take it; original reproduces the timing of the original stream, spacing deliveries to match the intervals at which messages first arrived, which is useful for realistic load reproduction.

1
js.CreateConsumer(ctx, "EVENTS", jetstream.ConsumerConfig{
2
    DeliverPolicy: jetstream.DeliverByStartTimePolicy,
3
    OptStartTime:  &startOfLastWeek,
4
    ReplayPolicy:  jetstream.ReplayInstantPolicy,
5
})

To rebuild a read model or projection, you do not coordinate with the producing systems or ask anyone to re-emit events. You create a fresh consumer, point it at the sequence or timestamp you want, replay the history JetStream already holds, and let it catch up to live. Any authorized consumer can rewind; the producers never know it happened.

FAQ

How do acknowledgments work in NATS JetStream? There are two. A publish acknowledgment (PubAck) confirms the server persisted your message to the stream per its storage and replication configuration. A consumer acknowledgment confirms a subscriber finished processing a delivered message; under an explicit ack policy the consumer must ack each message, and the durable cursor advances only past messages that are acknowledged, terminated, or bounded out by MaxDeliver.

How do I retry a failed message with a backoff? Either let the ack wait expire or nak with an explicit delay. The consumer’s BackOff schedule spaces redeliveries that happen because no ack arrived in time. A plain Nak bypasses it and redelivers immediately, and NakWithDelay uses the delay you pass instead of the schedule (on a consumer that also has BackOff set, later naks can wait longer than that delay, so pick one mechanism). To back off on a known transient error, either don’t ack (let BackOff drive the retry) or compute the delay yourself from the message’s delivery count and call NakWithDelay. Size MaxDeliver against the backoff schedule so the retries you want all get a chance to run.

Does NATS have a dead-letter queue? Not as a single configuration flag. You build one by bounding redelivery with MaxDeliver, capturing the resulting max-deliveries advisories into their own stream, and running a small handler that fetches each failed message by sequence and republishes it to a DLQ subject captured by a dedicated stream. The result is a durable, filterable, replayable dead-letter stream.

How do I stop a poison message from redelivering forever? Set MaxDeliver on the consumer to bound the number of delivery attempts, or call Term in your handler when you detect a permanent failure. Both stop redelivery; MaxDeliver also emits an advisory you can capture for a DLQ.

How do I replay past messages in JetStream? Create a consumer with a DeliverPolicy of all, by start sequence, or by start time, and choose a ReplayPolicy of instant or original timing. The consumer reads the retained history without involving the original producers.

Is JetStream exactly-once? JetStream provides exactly-once publish within the deduplication window via the Nats-Msg-Id header, and at-least-once delivery to consumers. End-to-end exactly-once processing is achieved the same way it is in any honest distributed system: build idempotent consumers on top of at-least-once delivery to get effectively-once results. Avoid treating unqualified “exactly-once” as a delivery guarantee.

Go deeper

• JetStream concepts, consumers, and replay in the official documentation at docs.nats.io.
• Runnable, client-by-client examples at natsbyexample.com, including acknowledgment and redelivery patterns.
• The NATS Architecture Decision Records for the design intent behind streams, consumers, and deduplication.

Two companion posts in this pack extend the picture: an architecture piece on whether industrial IoT deployments really need separate protocol brokers, and a concept piece arguing that message reliability belongs in the infrastructure rather than in application code. This post is the implementation those two point to.

Delivery guarantees in JetStream come down to a publish acknowledgment, durable cursors, three consumer verbs, bounded redelivery, advisories, and replay. Build the DLQ once from advisories and a get by sequence, and the absence of a one-line config reads less like a gap and more like a design decision.

Want help from the NATS experts? Meet with our architects to get help tailored to your use case and environment.