JetStream disaster recovery across two Kubernetes clusters

A community member asked how to run NATS with JetStream across a “live” Kubernetes cluster and a “DR” Kubernetes cluster while preserving stream and consumer state during failover.

The short answer: there is no simple active/passive switch that automatically preserves all JetStream stream and consumer state across two independent Kubernetes clusters. The right design depends on whether you need automatic failover, how much manual recovery you can tolerate, and whether preserving consumer position is required.

Why a supercluster is not enough for JetStream storage failover

A NATS supercluster connects multiple NATS clusters, but JetStream stream assets are still placed within a single cluster. A stream, including its replicas, is bound to one cluster rather than being automatically spread across all clusters in a supercluster.

That means a supercluster can be useful for multi-cluster connectivity, but it does not by itself make a JetStream stream’s replicated storage span both Kubernetes clusters. If the cluster that owns the stream data is unavailable, clients connected elsewhere in the supercluster should not be expected to continue reading that stream as if the data had been replicated there.

Option 1: Stretch one NATS cluster across both Kubernetes clusters

One possible design is to run a single NATS cluster whose servers are distributed across both Kubernetes clusters. For example, you might have three NATS servers in site A and three NATS servers in site B, all participating in one NATS cluster rather than two separate NATS clusters joined by a supercluster.

This can work, but it is operationally more complex than a normal single-Kubernetes-cluster deployment.

Routing requirements

For a single NATS cluster, servers need to be able to route to one another. The route list does not necessarily need to be complete on every server at startup if the cluster is configured correctly and servers advertise routable addresses, but each server must be able to discover and connect to enough peers.

In Kubernetes, that usually means you need to think carefully about:

• stable DNS names or addresses for NATS routes across both Kubernetes clusters;
• network connectivity between the clusters;
• the NATS cluster configuration, including advertised route addresses;
• how the Kubernetes deployment or Helm values render the final server configuration.

The standard Helm chart is convenient for common deployments, but cross-Kubernetes-cluster routing may require customization. If you cannot express the required route and advertise configuration cleanly through the chart values, a custom deployment or a carefully reviewed rendered configuration may be easier to reason about.

Quorum is the hard part

JetStream uses quorum-based replication. There is quorum for the JetStream metadata layer, and there is also quorum for each replicated stream and consumer.

That distinction matters during a site failure.

Consider a six-server NATS cluster split evenly across two Kubernetes clusters:

• 3 servers in Kubernetes cluster A
• 3 servers in Kubernetes cluster B

If one Kubernetes cluster goes down, only three of six NATS servers remain. For the cluster-wide metadata group, three out of six is not a majority. Without metadata quorum, administrative actions that require the metadata leader may not be possible.

Individual replicated streams and consumers have their own RAFT groups. JetStream caps a stream at five replicas, so even in a six-server cluster a stream cannot keep a replica on every server. For a stream with five replicas, quorum is three. Depending on where those five replicas were placed, some streams may still have three surviving replicas after a site failure while others may only have two surviving replicas. The latter will not have quorum.

So even if some data exists on the surviving side, the system may not be fully operational until quorum is restored.

Manual recovery may be required

In a two-site split, recovery may involve manual operations. A common shape is:

1. Start an additional NATS server in the surviving Kubernetes cluster so the metadata layer can regain quorum.
2. After a metadata leader exists, use the appropriate NATS administrative tooling — for example, the nats server cluster peer-remove command — to remove a failed server peer so affected RAFT groups can re-establish quorum.
3. Allow JetStream to repair and place replicas on available servers.

The exact commands and sequence should be tested for your deployment. Peer removal is not something to improvise during an outage. Also, removing failed peers has data-safety implications: you generally do not want to blindly remove all down peers, because those servers may still contain data and may later return.

The important point is that a stretched two-site cluster is not a magic active/passive design. It can be made to work for some environments, but it needs a tested runbook and a clear understanding of quorum behavior.

Option 2: Use stream sources or mirrors between clusters

Another design is to keep separate NATS clusters and use JetStream source or mirror functionality to copy stream data from one cluster to another.

This can provide a copy of stream messages in another cluster, but it has an important limitation: source and mirror replication does not replicate consumer state. If you fail over to the other cluster, consumers there may need to be recreated or resumed according to that cluster’s local state. If the consumer state was not preserved, applications may replay messages from an earlier point than expected.

For at-least-once systems, applications should already be prepared for duplicates. But replaying from the beginning, or from a different point than the original durable consumer, may be operationally significant.

Be careful with bidirectional sourcing

Bidirectional stream sourcing can also be tricky. If cluster A sources from cluster B and cluster B sources from cluster A on the same subjects, you need to avoid creating a message loop.

Subject transforms or careful subject partitioning may help, but that adds complexity. Before adopting this pattern, be explicit about:

• which cluster is authoritative for each subject;
• whether replication is one-way or bidirectional;
• how loops are prevented;
• how failback works after the original site returns;
• what consumer state is expected to look like after failover.

For many teams, sources and mirrors are better suited to copying data for replay, migration, analytics, or recovery scenarios that can tolerate some data loss than to seamless failover that preserves durable consumer progress exactly. Source and mirror replication is asynchronous, so any messages not yet copied when a cluster fails will not be present on the other side — this approach has a non-zero recovery point objective (RPO).

Streams and consumers follow the same placement constraints

Durable consumer state is part of the JetStream state that needs quorum and placement. By default, consumers follow the stream. If the stream is in one cluster, its consumers are not independently replicated into another cluster by a supercluster. Similarly, a stream mirror or source can copy stream messages, but it should not be treated as a full copy of all consumer positions.

This is often the key surprise in DR designs: preserving messages is not the same as preserving consumption progress.

Practical guidance

Before choosing a design, decide what you actually need to preserve:

• Messages only: stream sources or mirrors may be sufficient, provided your applications can tolerate replay and duplicate delivery.
• Messages plus consumer position: you likely need the original stream and consumer RAFT groups to remain available or be recovered, which brings quorum and peer-recovery operations into the design.
• Automatic site failover: a two-site, evenly split quorum system is difficult. Consider whether you need another failure domain, a different topology, or a more explicit manual recovery process.
• Operational simplicity: avoid overfitting the Kubernetes deployment mechanism. If the Helm chart makes cross-cluster route configuration hard to understand, a custom deployment may be easier to operate safely.

For any DR design, run failure drills outside production first. A local Docker-based test can be useful for learning the RAFT and peer-removal behavior before adding Kubernetes networking, storage, and service discovery to the mix.

Choosing the right topology

For JetStream, disaster recovery across two Kubernetes clusters is mainly about quorum and state placement. A supercluster does not automatically spread a stream’s storage across sites. A stretched single NATS cluster can preserve stream and consumer state, but it requires careful route configuration and a tested quorum-recovery plan. Stream sources and mirrors can copy messages to another cluster, but they do not provide a complete copy of consumer progress.

Choose the topology based on your RPO/RTO goals, whether consumer state must survive failover, and how much manual recovery you can safely operate during an outage.

---

Want help from the NATS experts? Meet with our architects to get help tailored to your use case and environment.