Scaling JetStream Last-Per-Subject Streams with Many Subjects

A common community question is how a JetStream stream behaves when it retains only the latest message per subject and the number of subjects grows very large.

The short answer: this pattern can work, but the details matter. With MaxMsgsPerSubject: 1, frequent updates create deleted messages inside the stream sequence range. Those are often called interior deletes. A large number of interior deletes can affect performance, especially for memory-backed streams and especially when subject update rates are uneven.

What MaxMsgsPerSubject: 1 actually does

A common configuration for “latest value per subject” looks like this conceptually:

1
Retention: Limits
2
Maximum Per Subject: 1
3
Maximum Age: optional TTL
4
Storage: Memory or File
5
Replicas: 3

With Maximum Per Subject: 1, JetStream keeps only the most recent message for each subject. When a new message arrives for a subject that already has a retained message, the older message for that subject is deleted.

For example:

1
seq 1: state.A
2
seq 2: state.B
3
seq 3: state.B
4
seq 4: state.B

If Maximum Per Subject is 1, the retained messages are:

1
state.A -> seq 1
2
state.B -> seq 4

The earlier state.B messages are deleted. If those deleted messages are between the stream’s first and last sequence, they are interior deletes.

This means a stream can have a relatively small number of currently retained messages while still having a large number of deleted messages recorded inside its sequence range.

Why hot subjects create interior deletes

Interior deletes are not just a function of the total number of subjects. They are strongly affected by the write distribution across subjects.

If every subject is written once, there are no per-subject replacements.

If one subject is written repeatedly while another subject is written rarely, the stream keeps only the latest message for the hot subject and marks the older versions as deleted.

For example:

1
state.A written once
2
state.B written 100 times

With MaxMsgsPerSubject: 1, state.A may still be retained at an early sequence, while state.B is retained at a much later sequence. The older state.B messages become deletes within the stream sequence range.

That delete pattern can become significant when you combine:

• high publish rate,
• many subjects,
• uneven subject update frequency,
• MaxMsgsPerSubject: 1,
• memory storage,
• replication,
• and long enough retention for the sequence range to grow.

Is publish latency expected to grow with the subject count?

Not directly in a simple “more subjects means slower publishes” way.

Latency depends on the storage implementation, stream settings, delete pattern, replication, message size, batching, consumers, and access pattern. However, if a memory-backed stream has a very large number of interior deletes, that can become a practical source of overhead.

A useful diagnostic clue is this combination:

• publish latency increases over time,
• nats stream info shows a high deleted-message count,
• the stream uses MaxMsgsPerSubject: 1,
• purging or recreating the stream temporarily restores latency,
• and the retained message count is much smaller than the total sequence churn.

That does not prove the deleted-message count is the only cause, but it is a strong reason to examine whether the stream design is creating too many deletes on the publish path.

Memory storage versus file storage

Memory-backed streams are attractive for low-latency workloads, but they are not always the best fit for large, long-lived, high-churn last-per-subject state.

A practical rule of thumb:

• Use memory storage for low-latency workloads with modest retained data and minimal interior deletes.
• Consider file storage for larger state, high subject cardinality, high update churn, or workloads that create many deletes.

File-backed JetStream streams are generally a better fit for large delete ranges and larger retained state. Also, file-backed does not necessarily mean every publish waits for a physical disk flush before acknowledgement. Recently written or read data can be served from cache, and writes may go through the operating system page cache. The exact latency profile still depends on your hardware, operating system, filesystem, replication, and workload, so benchmark your own path.

If your target is very low producer-to-live-consumer latency, test file-backed storage rather than assuming it will be too slow. For many workloads, the difference may be smaller than expected, while recovery and long-running behavior may be better.

A better shape for many “latest value” workloads

There are two related but different requirements that often get combined into one stream:

1. Live event delivery: consumers need new updates as they happen.
2. Current state snapshot: new consumers need the latest known value for every subject.

A single MaxMsgsPerSubject: 1 stream can serve both roles, but it also puts per-subject replacement work on the publish path.

An alternative design is to split the workload:

1. Keep the hot stream append-oriented

Use a limits-based stream for live updates, with retention controlled by age or bytes, and do not set MaxMsgsPerSubject: 1 on that hot stream.

Conceptually:

1
Stream: parsed-events
2
Storage: File or Memory, depending on workload
3
Retention: Limits
4
MaxAge: a few hours, or another operational window
5
MaxBytes: bounded
6
MaxMsgsPerSubject: unlimited

This avoids creating per-subject replacement deletes in the hot publish stream.

2. Maintain latest state separately

Use a separate state store for the latest value per key or subject. In NATS, a Key/Value bucket is often a natural fit for this because KV is built on JetStream and represents the latest value for a key.

Depending on your server and client support, you may be able to populate the KV bucket from the stream using stream sourcing and subject transforms, or you can update the KV bucket from an application-side processor.

This moves the “latest value” maintenance out of the critical producer publish path for the event stream.

Important caveat: a KV bucket is still backed by a JetStream stream. If the KV bucket is large, replicated, memory-backed, or high-churn, it can have its own storage and recovery considerations. For large latest-state buckets, file storage may also be the more practical choice.

What about DeliverLastPerSubject?

JetStream consumers support a deliver policy that can deliver the last message for each subject before continuing with new messages.

That can be useful when your stream already contains the latest messages needed by a new consumer. However, it does not remove the storage cost of retaining those last messages in the stream. If you require a snapshot for every subject even when some subjects have not updated recently, and you use MaxMsgsPerSubject: 1 to keep that snapshot, you can still create the same interior-delete pattern.

So DeliverLastPerSubject is useful, but it is not a complete substitute for choosing the right storage and retention model.

Do interior deletes in one stream affect another stream?

Streams are separate. A large number of interior deletes in one stream should not directly create the same publish-path overhead in another stream.

That said, streams still share server resources such as CPU, memory, disk, and network. If one stream causes heavy resource pressure, other workloads on the same cluster can still be indirectly affected. Operational isolation may require separate streams, accounts, clusters, or resource limits depending on your environment.

Watch recovery behavior during rolling restarts

Large replicated memory-backed streams can be more demanding during node restart and catch-up. A restarted node holds no persisted data for a memory-backed stream, so it must recopy the full stream state from the current leader, which can be heavy for large streams. If you see logs such as:

1
Catchup for stream ... stalled
2
stream catching up
3
catchup failed, too many retries

or if a stream appears inaccessible during recovery, treat that as an operational signal, not just a transient warning.

Things to check:

• stream size,
• storage type,
• replica count,
• whether the stream is memory-backed,
• whether the stream sources another stream or feeds a KV bucket,
• whether consumers or internal source consumers are taking a long time to create,
• and whether the issue reproduces at smaller versus larger retained sizes.

If file-backed storage avoids the issue in your environment, that is a strong reason to prefer it for that workload. If you can reproduce a catch-up stall reliably, collect a minimal reproduction, server version, stream configuration, logs, and operational steps before escalating.

Practical recommendations

For a workload publishing thousands of messages per second while retaining the latest value for many subjects:

1. Inspect deleted messages with nats stream info. A high deleted-message count with MaxMsgsPerSubject: 1 is an important clue.
2. Do not assume memory storage is always faster over time. Memory streams can be excellent, but high interior-delete churn may make file storage more predictable.
3. Test file-backed streams. File-backed streams cache recent reads and writes and may meet low-latency requirements while handling large delete patterns better.
4. Separate the hot log from latest-state storage when possible. Keep the publish stream append-oriented, and maintain the latest value in a separate KV bucket or processor-managed state stream.
5. Use DeliverLastPerSubject where it fits, but remember it does not eliminate the cost of retaining last-per-subject state.
6. Validate rolling restart behavior under realistic data sizes. Recovery characteristics matter as much as steady-state publish latency.
7. Shard only when it helps the model. Splitting subjects across multiple streams can reduce per-stream state and improve isolation, but it adds operational complexity.

Bottom line

For high-cardinality, high-churn “latest message per subject” workloads, the main scaling issue is often not the number of subjects alone. It is the combination of subject update distribution, MaxMsgsPerSubject: 1, retained sequence range, storage type, and replication.

If the stream accumulates many interior deletes, try file-backed storage and consider separating live event delivery from latest-state snapshots. Memory-backed streams are best reserved for workloads with modest retained state and minimal delete churn.

Want help from the NATS experts? Meet with our architects to get help tailored to your use case and environment.