NATS Consumer Sequence Ahead of Stream Sequence: Causes and Fixes

This check reports consumers with delivery history whose reported stream position is ahead of the last sequence observed for their stream. Most of the time nothing is wrong. Stream state and consumer state are sampled separately — even within a single server’s monitoring response — so on a busy stream the consumer can be sampled a moment after the stream, having already delivered messages the older stream snapshot doesn’t include yet. That is why Insights rates this finding as informational: on its own it doesn’t indicate lost data or corruption.

The finding is worth acting on only when it holds up against fresh state: the consumer’s delivered sequence stays ahead of the stream’s last sequence when you query both directly, and the gap doesn’t close as messages are published. That persistent form means the stream’s sequence space moved backwards underneath the consumer, and the consumer won’t deliver again until the stream catches up to its position.

Why this matters

The common case is a sampling effect. Normal publishing between the two samples moves the consumer past the older stream snapshot. The gap is small — roughly the publish rate multiplied by the sampling skew — it can recur across collection epochs on a hot stream, and it needs no action. Compare fresh stream and consumer state before considering any recovery step: deleting or resetting a healthy consumer because of this finding causes the disruption the check never reported.

The rare case is a stream that rolled back. If the stream’s last sequence genuinely sits below the consumer’s delivered position, the consumer is waiting for a sequence number the stream hasn’t reached. If the delivered sequence is 500,000 and the stream’s last sequence is 100,000, the consumer won’t process anything until 400,000 new messages are published — and if the stream was rebuilt or restored with fresh numbering, that may be a very long wait.

The persistent form is easy to miss. The consumer reports zero num_pending (from its perspective there are no new messages) and zero num_ack_pending, so dashboards show a consumer with no backlog while the stream accumulates messages nobody is processing. An explicit sequence comparison is the only way to see it.

From NATS Server 2.14.0, the server repairs one version of this by itself. When a non-replicated (R1) consumer recovers and finds its delivered sequence past the stream’s last sequence, it logs delivered sequence N past last stream sequence of M and reconciles the consumer’s state down to the stream, so new messages aren’t skipped. Replicated consumers aren’t reconciled this way.

Common causes

Sampling skew on an active stream (by far the most common). The stream and consumer positions Insights compares are captured at slightly different moments. Messages published and delivered in between put the consumer ahead of the stream snapshot. Nothing is wrong, and a direct query shows the stream’s last sequence at or beyond the consumer’s delivered sequence.

The remaining causes produce the persistent form, where the gap survives a fresh comparison:

Stream migration or movement across clusters. When a stream is moved to a different cluster via source-based replication, the new stream assigns its own sequence numbers, so consumers carried over from the original stream point far outside the new sequence range. (Mirrors preserve the origin’s sequence numbers, and backup/restore preserves the sequence state as of the snapshot — those paths cause this check through rollback, not renumbering.)
Raft state reset or leader election with data loss. If a Raft group loses quorum and is force-recovered, or if a leader election occurs while replicas were significantly behind, the stream’s last sequence may roll back. The consumer’s delivered sequence, persisted in its own Raft group, may not roll back in sync.
Stream deletion and recreation with the same name. An operator deletes a stream and recreates it with the same name but fresh data. Deleting the stream also deletes its consumers, but if consumer state is restored from a backup taken before the deletion — or consumers are recreated from a snapshot of old state — their positions reference the old stream’s sequence space.
Backup restore with sequence mismatch. Restoring a stream from an older backup resets its sequence state to the backup point. Consumers that continued processing after the backup was taken now have delivered positions ahead of the restored stream’s last sequence.
JetStream domain change or migration. Moving between JetStream domains can result in sequence renumbering. Consumers that aren’t recreated alongside the stream end up with stale positions in the new domain’s sequence space.

How to diagnose

Confirm the finding against fresh state

Start here. Query the stream and the consumer directly, one straight after the other, and repeat a few seconds later. If the stream’s last_seq is at or beyond the consumer’s delivered stream_seq — or the gap shrinks and closes as messages flow — the finding was sampling skew and there is nothing to fix. Continue only if the consumer stays ahead.

Compare consumer delivered sequence to stream last sequence

# Get the stream's current state
nats stream info ORDERS --json | jq '{
  first_seq: .state.first_seq,
  last_seq: .state.last_seq,
  messages: .state.messages
}'

# Get the consumer's delivered position
nats consumer info ORDERS my-consumer --json | jq '{
  stream_seq_delivered: .delivered.stream_seq,
  ack_floor_stream_seq: .ack_floor.stream_seq,
  num_pending: .num_pending
}'

If stream_seq_delivered is still greater than the stream’s last_seq on repeated fresh queries, the consumer is genuinely ahead of the stream. On an active stream, query the stream second: a stream read taken after the consumer read can only err towards hiding the condition, never towards inventing it.

Scan all consumers on a stream

STREAM="ORDERS"
LAST_SEQ=$(nats stream info "$STREAM" --json | jq '.state.last_seq')

for consumer in $(nats consumer list "$STREAM" --names); do
  DELIVERED=$(nats consumer info "$STREAM" "$consumer" --json | jq '.delivered.stream_seq')
  if [ "$DELIVERED" -gt "$LAST_SEQ" ]; then
    echo "AHEAD: consumer=$consumer delivered=$DELIVERED last_seq=$LAST_SEQ gap=$((DELIVERED - LAST_SEQ))"
  fi
done

Programmatic detection

1
import (
2
    "fmt"
3
    "github.com/nats-io/nats.go"
4
)
5

6
func checkSequenceAhead(js nats.JetStreamContext, streamName string) error {
7
    stream, err := js.StreamInfo(streamName)
8
    if err != nil {
9
        return err
10
    }
11
    lastSeq := stream.State.LastSeq
12

13
    for consumer := range js.ConsumerNames(streamName) {
14
        info, err := js.ConsumerInfo(streamName, consumer)
15
        if err != nil {
16
            continue
17
        }
18
        if info.Delivered.Stream > lastSeq {
19
            fmt.Printf("AHEAD: stream=%s consumer=%s delivered=%d last_seq=%d gap=%d\n",
20
                streamName, consumer, info.Delivered.Stream, lastSeq,
21
                info.Delivered.Stream-lastSeq)
22
        }
23
    }
24
    return nil
25
}

1
import asyncio
2
import nats
3

4
async def check_sequence_ahead(stream_name: str):
5
    nc = await nats.connect()
6
    js = nc.jetstream()
7

8
    stream = await js.stream_info(stream_name)
9
    last_seq = stream.state.last_seq
10

11
    # consumers_info returns up to 256 consumers per page; pass offset= to page further
12
    for info in await js.consumers_info(stream_name):
13
        if info.delivered.stream_seq > last_seq:
14
            gap = info.delivered.stream_seq - last_seq
15
            print(f"AHEAD: stream={stream_name} consumer={info.name} "
16
                  f"delivered={info.delivered.stream_seq} last_seq={last_seq} gap={gap}")
17

18
    await nc.close()
19

20
asyncio.run(check_sequence_ahead("ORDERS"))

Determine the root cause

Check whether the stream was recently recreated, restored, or migrated:

# Check stream creation time vs consumer creation time
nats stream info ORDERS --json | jq '.created'
nats consumer info ORDERS my-consumer --json | jq '.created'

If the consumer’s created timestamp predates the stream’s (impossible in normal operation, since deleting a stream also deletes its consumers), the stream was recreated and the consumer’s state was restored from a backup or external snapshot of the old deployment.

How to fix it

Only apply these fixes to the persistent form confirmed above. A finding caused by sampling skew needs no change, and resetting or recreating a healthy consumer discards its delivery state for nothing.

Delete and recreate the consumer

This is the most reliable fix. On servers older than 2.14.0, resetting requires deleting and recreating the consumer. On 2.14.0+ you can instead reset it in place with nats consumer reset (see below) — provided the consumer’s deliver policy is all, by_start_sequence, or by_start_time; consumers configured with last, new, or last_per_subject reject custom-sequence resets and still need delete-and-recreate:

# Record the consumer's configuration
nats consumer info ORDERS my-consumer --json | jq '.config' > /tmp/consumer-config.json

# Delete the stale consumer
nats consumer rm ORDERS my-consumer -f

# Recreate with appropriate deliver policy
nats consumer add ORDERS my-consumer \
  --pull \
  --deliver all \
  --ack explicit \
  --filter "orders.>" \
  --max-deliver 5 \
  --replay instant \
  --defaults

Choose the deliver policy based on your needs:

--deliver all — process every message currently in the stream
--deliver new — skip existing messages, only process future publishes
--deliver N (a literal message sequence number, e.g. --deliver 1000) — start from a specific stream sequence

Reset the consumer in place (server 2.14.0+)

On nats-server 2.14.0+ (with natscli 0.4.0+), you can reset the consumer’s position without deleting it:

nats consumer reset ORDERS my-consumer --sequence <first_seq>

--sequence N makes sequence N the next message delivered. Omitting --sequence resets to the consumer’s ack floor, which doesn’t help here — the entire state is ahead of the stream. The reset cannot go below the consumer’s configured start sequence or start time, and a custom --sequence is only accepted for consumers whose deliver policy is all, by_start_sequence, or by_start_time — consumers configured with last, new, or last_per_subject reject it. Those consumers (and any consumer on a server older than 2.14.0) must be deleted and recreated instead.

Fix the root cause

For stream migration: Always recreate consumers after migrating a stream to a new cluster. Consumer positions are tied to the source stream’s sequence space and are not transferable.

For backup/restore: Include consumer state in your backup/restore procedures. If you restore a stream, also restore or recreate its consumers to ensure sequence alignment.

For stream deletion/recreation: Deleting a stream also deletes its consumers. If consumers are managed declaratively (e.g., via Terraform or a CI/CD pipeline), ensure the pipeline recreates them after the stream is recreated rather than restoring their old state.

Prevent future occurrences

Pair stream operations with consumer validation. After any stream-level operation (purge, restore, migrate, recreate), run a validation pass that compares all consumer positions against the stream’s sequence range.

Monitor with Insights. Synadia Insights evaluates consumer-005 every collection epoch and reports it as an informational finding. A one-off finding on a busy stream is expected; the same consumer reported epoch after epoch following a restore, migration or recovery is the signal to run the fresh-state comparison above.

Frequently asked questions

Will the consumer self-heal if enough new messages are published?

Theoretically, yes — if the stream eventually reaches the consumer’s delivered sequence, the consumer would resume. But this requires publishing enough messages to close the gap, which could be millions of messages. In practice, if the gap was caused by a stream recreation or restore, the sequence numbering may never reach the consumer’s position. Don’t wait for self-healing; reset the consumer.

Does this affect both push and pull consumers?

Yes. Both consumer types maintain a delivered sequence cursor. In the persistent form, where that cursor is genuinely ahead of the stream, neither type will deliver messages. Pull consumers may be slightly more obvious — pull fetches will repeatedly time out with no messages — but push consumers will silently do nothing.

Can this happen during normal operation without any manual intervention?

The finding, yes — routinely. On an actively published stream, sampling skew between the stream and consumer readings produces it with nothing wrong. The persistent form is different: it almost always follows an administrative action (restore, migration, or deletion and recreation) or a recovery after data loss that rolled the stream’s sequences back.

Why is this check informational rather than critical?

Because the evidence it has can’t separate the two cases. Insights compares a consumer position and a stream position that were captured at different moments, so a consumer that is simply keeping up with a fast stream looks the same as one stranded by a rollback. The finding doesn’t indicate lost data or corruption by itself; it tells you where a fresh comparison is worth running.

What happens to messages published while a consumer is genuinely ahead?

Messages accumulate in the stream normally. They’re just not being delivered to that consumer. Once you reset the consumer (with deliver policy all), it will process all accumulated messages. If the stream has retention limits (max_msgs, max_bytes, max_age), messages may be evicted before the consumer is fixed — resulting in permanent message loss for that consumer.

How is this different from consumer-004?

consumer-004 fires when the consumer is behind the stream’s first sequence (pointing at deleted messages). consumer-005 fires when the consumer is ahead of the stream’s last sequence (pointing at messages that don’t exist yet). Both are informational, and both are usually benign: consumer-004 typically follows purges and retention eviction, after which the consumer simply skips forward, while consumer-005 is typically sampling skew. Only the persistent form of consumer-005 — after a stream migration, restore or sequence rollback — leaves a consumer waiting.

FEATURED

RESOURCES

Comparisons