NATS High Interior Deletes on Stream: Causes, Diagnosis, and Fixes

Interior deletes occur when messages in the middle of a JetStream stream’s sequence range are removed — as opposed to messages being trimmed from the beginning (by age or count limits) or the end. Every interior delete creates a “hole” in the stream’s sequence numbering. The server tracks these holes in an in-memory bitmap that must be loaded during server restart, leader election, and replica catch-up. When the count of interior deletes is very high — tens of thousands or more — this bitmap becomes a significant source of memory pressure and recovery latency.

Why this matters

JetStream streams maintain a contiguous sequence space. Message 1 is followed by message 2, then 3, and so on. When a message in the interior of this sequence is deleted (say, message 500 out of a stream containing messages 1–10,000), the server must remember that sequence 500 no longer exists. It records this in a deleted-sequence bitmap.

Under normal stream operations with time-based or count-based retention, messages are removed from the head of the stream. The first sequence simply advances, and no bitmap is needed. Interior deletes arise from specific operations: explicit message deletion via the API, per-subject limits in a multi-subject stream (where the oldest message for a specific subject is removed while newer messages for other subjects remain), or purge operations that target specific subjects.

The memory cost scales with the number of deleted sequences, not the number of remaining messages. A stream with 1 million messages and 500,000 interior deletes requires the bitmap to track every one of those 500,000 gaps. This bitmap is held entirely in memory and cannot be paged to disk. During server startup, the stream recovery process reads the entire bitmap before the stream becomes available. During Raft leader election or replica catch-up, the bitmap is transmitted to the new leader or catching-up replica as part of the stream state snapshot.

In extreme cases, a stream with hundreds of thousands of interior deletes can take minutes to recover on server restart, during which time all consumers on that stream are unavailable. Replica catch-up after a transient network partition transfers megabytes of bitmap data, adding to the time before the cluster is fully healthy. And the ongoing memory consumption — often the most opaque part — silently reduces the server’s capacity for other JetStream assets.

Common causes

• Per-subject limits on multi-subject streams. This is the most common cause. A stream configured with max_msgs_per_subject removes the oldest message for a subject when the limit is reached. If the stream has many subjects with interleaved messages, each per-subject eviction creates an interior delete. Over time, the count grows proportionally to message volume.

• Explicit message deletion via API. Applications that delete specific messages (by sequence number) after processing — for example, removing PII after a retention period or cleaning up completed work items — generate interior deletes. This pattern is especially problematic when deletions are sparse and spread across a wide sequence range.

• Subject-scoped purge operations. Purging a specific subject from a multi-subject stream (nats stream purge STREAM --subject "orders.completed") removes all messages matching that subject, leaving gaps among messages for other subjects.

• Interest-based retention with multiple consumers. With interest retention policy, messages are removed once all defined consumers have acknowledged them. If consumers process at different rates, messages acknowledged by all consumers are removed from the interior of the sequence space while messages still pending for at least one consumer remain.

• Republish or migration patterns. Workflows that selectively delete messages after republishing them to a new stream create interior deletes in the source stream. If the migration processes messages out of order, the deletes are scattered across the sequence range.

How to diagnose

Check stream delete count

nats stream info STREAM --json | jq '{
  name: .config.name,
  messages: .state.messages,
  first_seq: .state.first_seq,
  last_seq: .state.last_seq,
  num_deleted: .state.num_deleted,
  sequence_span: (.state.last_seq - .state.first_seq + 1),
  delete_ratio: (if (.state.last_seq - .state.first_seq + 1) > 0 then (.state.num_deleted / (.state.last_seq - .state.first_seq + 1) * 100 | round) else 0 end)
}'

Key indicators:

• num_deleted — The total count of interior deletes. Values above 100,000 warrant attention; above 500,000 is a significant concern.
• delete_ratio — Interior deletes as a percentage of the total sequence span. Ratios above 30–40% indicate the bitmap is a major component of the stream’s memory footprint.

Find streams with high interior deletes

nats stream ls --json | jq -r '.[] | select(.state.num_deleted > 100000) | "\(.config.name)\t\(.state.num_deleted)\t\(.state.messages)"'

Check per-subject limits

If the stream has max_msgs_per_subject configured, that’s likely the source:

nats stream info STREAM --json | jq '{
  max_msgs_per_subject: .config.max_msgs_per_subject,
  num_subjects: .state.num_subjects,
  estimated_deletes_per_cycle: .state.num_subjects
}'

A stream with 10,000 subjects and max_msgs_per_subject: 1 will generate roughly 10,000 interior deletes per full cycle of message production across all subjects.

Monitor recovery time

During a server restart, check the logs for stream recovery duration:

grep -i "stream.*recovered\|stream.*loaded" /var/log/nats/nats-server.log

Streams with high interior delete counts will show noticeably longer recovery times compared to similar-sized streams without interior deletes.

Programmatic check

1
package main
2

3
import (
4
  "context"
5
  "fmt"
6
  "github.com/nats-io/nats.go/jetstream"
7
)
8

9
func checkInteriorDeletes(js jetstream.JetStream, threshold uint64) error {
10
  ctx := context.Background()
11
  streams := js.StreamNames(ctx)
12
  for name := range streams.Name() {
13
    stream, err := js.Stream(ctx, name)
14
    if err != nil {
15
      continue
16
    }
17
    info, err := stream.Info(ctx)
18
    if err != nil {
19
      continue
20
    }
21
    if info.State.NumDeleted > threshold {
22
      fmt.Printf("WARN: stream %s has %d interior deletes (messages=%d, span=%d)\n",
23
        info.Config.Name,
24
        info.State.NumDeleted,
25
        info.State.Msgs,
26
        info.State.LastSeq-info.State.FirstSeq+1,
27
      )
28
    }
29
  }
30
  return nil
31
}

How to fix it

Immediate: purge to reset the delete bitmap

If the stream can tolerate data loss (or the messages have already been consumed), purging the stream resets the sequence space and clears the delete bitmap entirely:

nats stream purge STREAM

For streams where you need to keep recent data, purge messages older than a safe window:

nats stream purge STREAM --keep 10000

This keeps the most recent 10,000 messages and removes everything else — including the interior deletes in the purged range.

Short-term: restructure retention to avoid interior deletes

Replace per-subject limits with dedicated streams. Instead of one stream with max_msgs_per_subject: 1 across 10,000 subjects, create separate streams (or use a KV store, which is purpose-built for this pattern):

# If you're using max_msgs_per_subject for latest-value semantics,
# a KV store is the right abstraction
nats kv add my-state --history 1 --storage file --replicas 3

Use time-based retention instead of per-subject limits. If the goal is to keep messages for a fixed duration, max_age trims from the head of the stream without creating interior deletes:

nats stream edit STREAM --max-age 24h --max-msgs-per-subject=-1

Prefer limits retention over interest retention. Interest-based retention inherently creates interior deletes when consumers process at different rates. If you can switch to limits retention with appropriate max_age or max_msgs, the stream trims from the head and avoids interior gaps.

Long-term: design streams to minimize interior deletes

Separate high-cardinality subjects into their own streams. A stream with 100,000 subjects and per-subject limits is a recipe for interior deletes. Group related subjects into smaller streams where per-subject limits produce fewer scattered deletions.

Use compaction-friendly patterns. If you need latest-value-per-key semantics (which is what max_msgs_per_subject: 1 provides), use a NATS KV store. KV stores are backed by JetStream but are designed and optimized for this access pattern, including more efficient compaction.

Schedule periodic stream compaction. For streams that inevitably accumulate interior deletes, implement a maintenance window where the stream is purged and repopulated from a source-of-truth (another stream, a database, or a mirror). Note that nats stream copy only clones the configuration, not the data — to actually compact, set up a fresh stream and re-source the data:

# 1. Capture the old stream's config so you can recreate the schema
nats stream info OLD_STREAM --json | jq '.config' > /tmp/old-config.json

# 2. Create the replacement stream with the same config under a new name
nats stream add NEW_STREAM --config /tmp/old-config.json

# 3. Replay live data into NEW_STREAM from your source of truth (mirror,
#    upstream service, or a sourced configuration). Then swap consumers to
#    NEW_STREAM and delete OLD_STREAM:
nats stream rm OLD_STREAM -f

Synadia Insights monitors interior delete counts across all streams, flagging those that cross configurable thresholds before the memory and recovery impact becomes critical.

Frequently asked questions

How much memory does the delete bitmap actually use?

The bitmap is stored as a compressed set of sequence ranges. For contiguous deletes (e.g., sequences 100–5,000 all deleted), the representation is compact — just the range start and end. For scattered deletes (every other sequence), each gap requires its own entry. In the worst case — fully interleaved deletes — the bitmap approaches roughly 8–16 bytes per deleted sequence. A stream with 1 million scattered interior deletes could consume 8–16 MB of memory just for the bitmap.

Will the bitmap shrink if I delete more messages?

Deleting additional messages in the interior adds to the bitmap — it doesn’t shrink it. The bitmap only shrinks when the stream’s first sequence advances past deleted regions (i.e., when head truncation catches up to the delete range). Purging or advancing max_age/max_msgs to remove the oldest portion of the stream is the way to reclaim bitmap memory.

Can I see the actual deleted sequences?

Yes. The stream info API with the deleted_details option returns the list of deleted sequence numbers:

nats stream info STREAM --json | jq '.state.deleted'

For streams with very high delete counts, this list can be large. Use it for debugging, not routine monitoring.

Does this affect stream mirrors and sources?

Yes. Mirrors and sources replicate the stream state, including the delete bitmap. A mirror of a stream with high interior deletes inherits the same memory footprint and recovery characteristics. If you mirror a stream to reduce the delete count, you need to purge the mirror independently — it doesn’t get a fresh sequence space from mirroring.