Should You Shard JetStream Streams with Many Durable Consumers?

A community member asked whether it is good practice to shard JetStream streams, subjects, or consumers when a single stream starts approaching a large number of durable pull consumers, such as 10,000.

Short answer

In general, you should plan for sharding or another design change before a single JetStream stream becomes responsible for a very large number of independent durable consumers.

That does not mean you must immediately shard the moment you see a specific consumer count. The right decision depends on your workload, message size, storage, replica count, retention settings, consumer lag, hardware, and operational tolerance for migration risk.

A practical position is:

If the system is healthy, consumers stay near the head of the stream, and you have CPU and I/O headroom, a single stream may continue to be acceptable.
If consumers are independent, numerous, frequently lagging, or expected to grow significantly, sharding should be considered part of the design rather than a last-minute rescue.
If sharding would be a breaking change, benchmark and monitor carefully, but also start designing a migration path before the system is under pressure.

Why one stream does not scale linearly forever

A JetStream stream is a unit of storage and replication. Adding more servers to a cluster can improve overall system capacity, but a single stream is still not the same as many independent partitions spread across the system. A replicated stream is coordinated by one leader at a time, so replication is primarily about availability and consistency for that stream, not unlimited horizontal throughput for it.

Durable consumers also have state. For each durable consumer, the system must track information such as where that consumer is in the stream and what work is pending or acknowledged. With a small number of consumers, that overhead is usually not the dominant cost. With thousands or tens of thousands of independent durable consumers, the background work becomes more significant.

The important point is that this overhead does not scale linearly. Server-side handling of many consumers has been optimized over time, but performance still does not grow linearly with consumer count. A design that looks fine with 10 durable consumers may behave differently with 10,000, especially if those consumers are not all reading at roughly the same pace.

Pull consumers help, but they do not remove consumer state

Durable pull consumers are often a good fit when clients want to control when they fetch work. They avoid some of the delivery behavior associated with push consumers, and they can be very useful for worker-style applications.

However, durable pull consumers are still durable consumers. Each one still has server-side state. If every application instance, user, tenant, device, or integration gets its own durable consumer, the system may eventually spend a meaningful amount of CPU and coordination effort just keeping track of all of those independent positions.

If many consumers are effectively doing the same work, consider whether they really need separate durable state. Multiple worker instances can bind to and fetch from a single shared pull consumer, which distributes messages across those workers without giving each one its own durable position. If each consumer truly needs an independent view of the stream, then sharding becomes a more relevant tool.

What does the JetStream cache behavior mean?

JetStream writes stream data to storage. Recently written messages may also remain available in memory or operating system cache for a short period. When consumers read messages soon after they are written, the server may be able to serve those messages without going back to disk.

If consumers fall behind, the situation changes. Once messages are no longer available from cache, serving those delayed consumers can require additional disk reads. At small scale this may be unnoticeable. At large scale, many delayed consumers rereading older data can add significant I/O pressure.

A useful mental model is:

Consumers near the head of the stream are usually cheaper to serve.
Consumers that are seconds, minutes, or hours behind can be more expensive.
Many lagging independent consumers can turn a mostly sequential write workload into a workload with much more read pressure.

The exact cache behavior depends on the server, operating system, workload, and deployment. Do not rely on a fixed cache window as a contract. Instead, test the delayed-consumer case that your application may actually produce.

Signals that sharding may be needed

There is no universal number of durable consumers where a single stream becomes wrong. Instead, watch for the symptoms that indicate the stream or its consumers are becoming the bottleneck.

Useful signals include:

CPU saturation on the servers responsible for the stream.
Increasing JetStream API latency or pull fetch latency.
Consumers falling farther behind over time.
Increasing redeliveries or timeout-related behavior.
Growing disk read activity even when publish throughput is stable.
Disk I/O saturation or elevated storage latency.
Stream or consumer state updates becoming a noticeable source of load.
Operational difficulty recovering after restarts, failover, or bursts of lagging consumers.

If your benchmark shows I/O is fine today, that is useful evidence. But make sure the benchmark includes the failure modes you care about: slow consumers, delayed consumers, reconnect storms, realistic message sizes, realistic retention, and the same durability settings you use in production.

Common sharding approaches

Sharding does not have to mean one specific architecture. Common approaches include:

Subject partitioning

Instead of placing all messages on one subject pattern backed by one stream, partition the subject space. For example, route messages by tenant, region, customer, hash bucket, or workload class.

This can let you create multiple streams, each responsible for a subset of subjects. It also gives you a clearer path to placing different workloads on different resources.

Multiple streams

Create multiple streams that each hold part of the data. This can help distribute storage, replication, and consumer state. It can also limit the blast radius of a hot shard or a large group of lagging consumers.

The tradeoff is application complexity. Producers and consumers need to know how messages are assigned to streams, and operations teams need to monitor more moving parts.

Fewer durable consumers

Sometimes the better answer is not more streams, but fewer independent durable consumers. If many consumers are equivalent workers, they may be able to share a work distribution pattern instead of each maintaining an independent durable position.

This is workload-specific. If each tenant or user must independently replay and acknowledge messages, reducing consumer count may not be acceptable. But if the consumers are just competing workers, separate durable consumers may be unnecessary overhead.

Larger isolation boundaries

At some scale, the right boundary may be a separate account, domain, or cluster. That is a larger operational decision and should be driven by isolation, ownership, capacity, and failure-domain requirements, not only by consumer count.

When it is reasonable not to shard yet

If sharding is a breaking change, it is reasonable to avoid doing it prematurely. A single-stream design can be acceptable when:

Current throughput and latency are healthy.
Consumers remain close to the head of the stream.
CPU and disk I/O have substantial headroom.
Consumer count growth is understood and bounded.
You have monitoring that will show lag, latency, CPU, and storage pressure before users are affected.
You have a migration plan if the workload changes.

The risk is waiting until the stream is already saturated. Sharding under production pressure is usually harder than adding a shard-aware abstraction earlier.

What to benchmark

A benchmark should look like the production workload you are worried about, not only the happy path.

Include tests for:

The expected number of durable pull consumers.
The expected publish rate and message size.
The expected retention policy and storage type.
The same replication factor you plan to run.
Consumers that read immediately.
Consumers that are delayed by more than a few seconds.
Consumers that are delayed by minutes or longer, if that can happen.
Bursts of reconnecting or catching-up consumers.
Server restart or failover scenarios, if high availability is required.

A benchmark where all consumers stay perfectly caught up may miss the cost that appears when consumers drift behind and older messages must be read again from storage.

Practical guidance

For a large number of durable pull consumers on one stream, the main question is not whether 10,000 is always too many. The better question is whether the architecture assumes one stream can carry unbounded independent consumer state.

Use this decision framework:

Measure the current system under realistic load.
Test delayed and lagging consumers, not just current steady state.
Watch CPU, disk reads, latency, consumer lag, and redeliveries.
If consumers are independent and expected to grow, design a sharding path.
If many consumers do not need independent durable state, simplify the consumer model first.

A single stream can be a good starting point, but it is not a substitute for partitioning when the workload grows into many independent, lagging, or high-volume consumers. Shard when the operational and performance risks of centralizing everything in one stream become greater than the migration cost.

Want help from the NATS experts? Meet with our architects to get help tailored to your use case and environment.