NATS Large Deduplication Window: Causes, Diagnosis, and Fixes

The deduplication window on a JetStream stream defines how long the server retains message IDs for duplicate detection. When a publisher sets the Nats-Msg-Id header, the server checks the dedup map to prevent storing the same message twice. Each entry in this map consumes approximately 130–150 bytes of memory. With a large deduplication window and high message rates, the map can grow to gigabytes — and unlike most Go allocations, this memory is often not returned to the operating system. Reducing the deduplication window to match your actual publisher retry interval is the most effective way to control this memory consumption.

Why this matters

JetStream deduplication is a critical feature for exactly-once publishing semantics. When a publisher sends a message with a Nats-Msg-Id header, the server stores a mapping from that ID to the message’s stream sequence number. If the same Nats-Msg-Id arrives again within the deduplication window, the server returns the original sequence number instead of storing a duplicate. This handles the common case where a publisher retries after a timeout without knowing whether the original message was stored.

The dedup map lives entirely in memory. There is no disk-backed or LRU eviction mechanism — every message published within the window has an active entry in the map. The default deduplication window is 2 minutes, which is reasonable for most workloads. But if the window is set to an hour, a day, or longer, the map accumulates entries proportional to the publish rate multiplied by the window duration.

The math is straightforward but alarming at scale. A stream receiving 10,000 messages per second with a 1-hour deduplication window holds 36 million entries in the dedup map. At ~140 bytes per entry, that’s approximately 5 GB of memory — just for deduplication, before counting the actual message storage. With UUID-based message IDs (36 bytes each), the per-entry cost is at the high end of the range.

Worse, Go’s runtime garbage collector does not reliably release this memory back to the operating system. The dedup map is a long-lived allocation with constant churn (old entries expire, new entries are added). The Go runtime tends to hold onto this memory even after the map shrinks, because the freed heap pages are fragmented across the virtual address space. In practice, a server that once hit 8 GB of dedup map usage may never return below 6 GB even after the window expires — until the server process is restarted.

This creates an operational trap: the dedup map grows during traffic spikes, the memory is never reclaimed, and the server’s baseline memory consumption ratchets upward over time. Eventually, the server hits JetStream memory limits or OS memory limits, causing JetStream to stop accepting new data or the process to be OOM-killed.

Common causes

• Copy-pasted configuration with overly conservative windows. Teams set a large deduplication window “just to be safe” without calculating the memory impact. A 24-hour window seems reasonable until you multiply it by the message rate.

• Confusion between deduplication and retention. Some operators conflate the deduplication window with message retention. They set a long window thinking it controls how long messages are kept. In reality, retention is controlled separately by max_age, max_msgs, and max_bytes. The dedup window only controls how long the server remembers message IDs for duplicate rejection.

• Publisher retry intervals that don’t match the window. The dedup window only needs to be longer than the publisher’s maximum retry interval. If a publisher retries for at most 30 seconds before giving up, a 2-minute dedup window provides generous coverage. A 1-hour window provides no additional protection but consumes 30x more memory.

• Migrated configuration from other messaging systems. Teams migrating from Kafka or RabbitMQ may carry over deduplication settings from those systems, which have different memory characteristics. Kafka’s idempotent producer state is stored per-partition on disk; NATS dedup is entirely in-memory.

• Streams with the default window that have grown in throughput. Even the default 2-minute window can become expensive if the stream’s publish rate grows from 100 msg/s to 100,000 msg/s without anyone revisiting the dedup configuration.

How to diagnose

Check deduplication window on streams

nats stream info STREAM --json | jq '{
  name: .config.name,
  duplicate_window_ns: .config.duplicate_window,
  duplicate_window_human: (.config.duplicate_window / 1000000000 | tostring + "s"),
  messages: .state.messages,
  bytes: .state.bytes
}'

The duplicate_window is reported in nanoseconds. Convert to a human-readable duration and evaluate against the stream’s publish rate.

Find all streams with large windows

nats stream ls --json | jq -r '.[] | select(.config.duplicate_window > 120000000000) | "\(.config.name)\t\(.config.duplicate_window / 1000000000)s"'

This lists streams with dedup windows exceeding 2 minutes (120 seconds).

Estimate dedup map memory usage

Calculate the expected memory cost:

# Get the publish rate (messages per second)
nats server report jetstream --json | jq '.'

# Memory estimate:
# entries = publish_rate * window_seconds
# memory_bytes = entries * 140
# Example: 5000 msg/s * 3600s window = 18M entries * 140 bytes = ~2.5 GB

Check server memory usage

nats server report server --json | jq '.[] | {name: .name, mem: .mem, cpu: .cpu}'

If a server’s memory consumption is significantly higher than the aggregate JetStream storage would suggest, the dedup map is a likely contributor.

Programmatic detection

1
package main
2

3
import (
4
  "context"
5
  "fmt"
6
  "time"
7

8
  "github.com/nats-io/nats.go/jetstream"
9
)
10

11
func checkDedupWindow(js jetstream.JetStream, maxWindow time.Duration) error {
12
  ctx := context.Background()
13
  streams := js.StreamNames(ctx)
14
  for name := range streams.Name() {
15
    stream, err := js.Stream(ctx, name)
16
    if err != nil {
17
      continue
18
    }
19
    info, err := stream.Info(ctx)
20
    if err != nil {
21
      continue
22
    }
23
    if info.Config.Duplicates > maxWindow {
24
      fmt.Printf("WARN: stream %s has dedup window %v (max recommended: %v)\n",
25
        info.Config.Name, info.Config.Duplicates, maxWindow)
26
    }
27
  }
28
  return nil
29
}

How to fix it

Right-size the deduplication window

Set the dedup window to the minimum necessary for your publisher’s retry behavior. The formula is:

1
dedup_window = publisher_max_retry_duration + safety_margin

If your publisher retries for up to 10 seconds before giving up, a 30-second dedup window is generous. If it retries for up to 60 seconds, a 2-minute window covers it.

nats stream edit STREAM --dupe-window 30s

For streams where deduplication is not needed (publishers don’t set Nats-Msg-Id), the dedup map isn’t populated regardless of the window setting — but it’s still good practice to keep the window small.

Apply across all streams

# Find and fix all streams with windows over 2 minutes
for stream in $(nats stream ls --json | jq -r '.[] | select(.config.duplicate_window > 120000000000) | .config.name'); do
  echo "Updating $stream..."
  nats stream edit "$stream" --dupe-window 2m --force
done

Reclaim memory after reducing the window

Reducing the dedup window takes effect for new entries, but existing entries remain until they expire under the old window. To reclaim memory immediately:

1. Wait for the old window to expire. If you reduced from 1 hour to 2 minutes, wait 1 hour for all old entries to age out.
2. Restart the server. This is the most reliable way to reclaim memory held by Go’s runtime. After restart, the dedup map rebuilds with only the entries within the new window.

# Graceful server restart to reclaim memory
nats-server --signal quit  # SIGTERM equivalent for graceful shutdown --help

Optimize message IDs

If you must keep a longer dedup window, optimize the message ID format to reduce per-entry memory:

• Use short, sequential IDs instead of UUIDs. A 12-character base62 ID uses 12 bytes versus a UUID’s 36 bytes.
• Use publisher-scoped prefixes. IDs like pub1-00001 are shorter and more compressible than 550e8400-e29b-41d4-a716-446655440000.

1
// Instead of UUIDs:
2
msg.Header.Set("Nats-Msg-Id", uuid.New().String()) // 36 bytes
3

4
// Use compact sequential IDs:
5
msg.Header.Set("Nats-Msg-Id", fmt.Sprintf("%s-%d", publisherID, seq)) // ~15 bytes

Long-term: monitor dedup memory impact

Set up alerting on the combination of dedup window size and publish rate.

Synadia Insights evaluates deduplication window configuration against active message flow rates, alerting when the estimated in-memory dedup map poses a saturation risk — before it manifests as memory pressure or OOM events.

Frequently asked questions

Does the dedup window matter if publishers don’t set Nats-Msg-Id?

If no publisher sets the Nats-Msg-Id header, the dedup map is never populated. The window setting exists but has no memory impact. However, reducing the window to a small value is still recommended as a safeguard — if a publisher starts using message IDs in the future, a large window would immediately begin consuming memory.

Can I disable deduplication entirely?

There is no explicit “disable dedup” setting. Setting the window to the minimum value (typically 0 or a very small duration) effectively disables it. In practice, setting it to 1–2 seconds is safe for most workloads and keeps the dedup map negligibly small.

Why doesn’t Go release the memory after the map shrinks?

Go’s garbage collector frees objects from the heap, but the runtime doesn’t immediately return freed heap pages to the operating system. This is by design — reclaiming and re-requesting virtual memory pages has a cost. Go’s MADV_FREE / MADV_DONTNEED behavior means the RSS may stay elevated even though the Go heap has shrunk. The memory is “available” to the Go process for future allocations but isn’t visible as free to the OS. A process restart is the definitive way to return this memory.

What’s the default deduplication window?

The default is 2 minutes (120 seconds). This is set at the JetStream server level and applies to any stream that doesn’t explicitly configure its own window. For moderate publish rates (under 10,000 msg/s), the 2-minute default is reasonable. For high-throughput streams, even 2 minutes may warrant review — 10,000 msg/s × 120s × 140 bytes = ~168 MB per stream.

Does the dedup window affect stream mirroring or sourcing?

The dedup window applies to the stream receiving publishes, not to mirrors or sources. Mirrors replicate messages from the source stream and do not perform their own deduplication. If a message was deduplicated at the source, the mirror only sees the single stored message. However, if a mirror is also configured as a writable stream accepting direct publishes, its own dedup window setting applies to those direct publishes.