NATS Meta Quorum Lost

The meta cluster has lost quorum — enough peers are offline that the remaining servers cannot form a Raft majority. Without meta quorum, the JetStream API is completely stalled: no stream creation, no consumer creation, no configuration changes, and no placement decisions. This is a cluster-wide JetStream control plane outage.

Why this matters

The meta cluster is the brain of JetStream. It manages the registry of all streams and consumers, handles placement decisions (which server hosts which replica), processes every JetStream API request, and coordinates configuration changes across the cluster. All of this runs on Raft consensus, which requires a quorum — a strict majority of peers — to commit any operation.

When quorum is lost, no new Raft entries can be committed. The meta leader (if one was elected) cannot make progress because it can’t get acknowledgments from enough peers. In practice, this means the JetStream API hangs. Any client calling js.AddStream(), js.AddConsumer(), js.StreamInfo(), or similar operations will time out. Kubernetes operators, CI/CD pipelines, and automation that creates JetStream resources all stall. The cluster cannot self-heal — it can’t rebalance replicas, reassign leaders, or respond to topology changes.

The critical nuance: existing data streams may continue operating. If a stream’s own Raft group (separate from the meta group) still has quorum — say two of three stream replicas are on servers that are still running — that stream can still accept publishes and deliver messages to consumers. But you can’t modify it, can’t create new consumers for it, and can’t query its full state through the JetStream API. The system is in a read/write-for-existing-streams-only mode with a frozen control plane. This is a dangerous state because any additional failure (a stream losing its own quorum, a consumer needing reassignment) cannot be handled.

Common causes

• Multiple servers down simultaneously. The most direct cause. In a three-node cluster, losing two servers means only one remains — well below the quorum of two. This can happen during aggressive rolling restarts, concurrent hardware failures, or a shared infrastructure failure (same rack, same power circuit, same hypervisor host).

• Network partition splitting the cluster below quorum. A network failure isolates enough servers that no group of peers can form a majority. In a three-node cluster, a partition that isolates each server individually means no group has two peers — quorum is impossible on any side.

• Aggressive rolling restart. The most common operational cause. If you restart server A before server B (which was restarted earlier) has fully rejoined the meta group, you have two servers simultaneously out of the meta group. In a three-node cluster, that’s quorum lost. This happens when restart scripts don’t wait for meta group health between restarts.

• Shared infrastructure failure. Two of three servers running on the same cloud availability zone, hypervisor host, or network switch. When that shared resource fails, multiple servers go down simultaneously. This defeats the purpose of replication.

• Cascading resource exhaustion. One server goes down due to disk full. The increased load on the remaining servers causes a second server to hit resource limits (memory, CPU, disk I/O), and it either crashes or falls out of the meta group due to missed Raft heartbeats.

How to diagnose

Confirm meta quorum is lost

nats server report jetstream

The Raft Meta Group table shows each peer’s state. Count the servers showing Offline: true. If offline peers >= ⌈(cluster_size)/2⌉, quorum is lost. For a three-node cluster, two offline peers mean no quorum. For five nodes, three offline means no quorum.

If the nats CLI can’t connect to any server, or JetStream requests time out, that itself is strong evidence of quorum loss.

Check the health endpoint

curl -s http://localhost:8222/healthz

When meta quorum is lost, the health endpoint returns an error indicating JetStream is unavailable. The response includes specific details about the failure.

For a targeted JetStream health check:

nats server check jetstream

This will report the meta group state and whether quorum is maintained.

Identify which servers are down

nats server list

Compare the listed servers against your expected cluster membership. Missing servers are the ones that need to be restored.

If you can’t reach the cluster via the nats CLI, check each server’s monitoring endpoint directly:

# Check each server individually
curl -s http://<server-1>:8222/healthz
curl -s http://<server-2>:8222/healthz
curl -s http://<server-3>:8222/healthz

Check why servers are down

For each offline server:

# Is the process running?
systemctl status nats-server

# Check for OOM kills
dmesg | grep -i "out of memory"

# Check disk space
df -h /path/to/jetstream/store

# Check server logs for the shutdown reason
journalctl -u nats-server --since "1 hour ago" | tail -50

How to fix it

Immediate: restore quorum

Priority one: bring servers back online. This is the only way to restore meta quorum. Every second without quorum is a second where the JetStream control plane is frozen:

# Start the NATS server
systemctl start nats-server

# Verify it's running
systemctl status nats-server

# Check if it rejoined the meta group
curl -s http://localhost:8222/jsz | jq '.meta_cluster'

You need to restore enough servers to reach quorum. For a three-node cluster, bring at least one of the two offline servers back. For five nodes, restore at least one of the three offline servers (to get back to three of five).

If a server can’t start due to disk full:

# Free space — remove old logs, temp files, etc.
# DO NOT delete JetStream store data unless you understand the consequences
journalctl --vacuum-time=1h
find /tmp -type f -mtime +1 -delete

# Then start the server
systemctl start nats-server

If a server is permanently lost, add a replacement. Deploy a new NATS server with the same cluster configuration and JetStream enabled. It will join the meta group as a new peer. The existing meta leader will send it a snapshot, and it will catch up on Raft state:

1
jetstream {
2
  store_dir: /data/jetstream
3
  max_file_store: 100G
4
}
5

6
cluster {
7
  name: "production"
8
  routes: [
9
    "nats://server-1:6222"
10
    "nats://server-2:6222"
11
    "nats://server-3:6222"
12
  ]
13
}

If quorum can still be formed (e.g., you lost one of five servers and a majority remain online), remove the peer entry directly:

nats server cluster peer-remove <lost-server-name>

This adjusts the meta group size downward, reducing the quorum requirement. In a three-node cluster where one server is permanently lost, removing the peer makes it a two-node group (quorum = 2) — which means the remaining two servers can form quorum. But a two-node meta group has zero fault tolerance, so add a replacement server promptly.

Only ever peer-remove a server that is actually stopped. A running server that gets peer-removed reacts by turning JetStream off on itself, so confirm the “lost” server is truly down first.

If no leader can be elected at all — too many servers are permanently gone for the survivors to form a majority — peer-remove can’t help, because removing a peer is itself a change the meta group has to agree on, and there’s no quorum to agree with. From NATS 2.15, nats server cluster rescue is the supported way out. It temporarily lowers the quorum the meta group needs so the survivors can elect a leader again:

nats server cluster rescue

The command reports each server’s view before changing anything:

1
Current Cluster State
2

3
   Server   Cluster  Size  Peers               Quorum Requires  Rescuing
4
   n1-east  east     5     2 online / 5 peers  3                false
5
   n2-east  east     5     2 online / 5 peers  3                false

It then asks how many peers to require (defaulting to the number still online), broadcasts the change, and confirms per server:

1
Server "n1-east" now requires 2 quorum size (was 3)
2
Server "n2-east" now requires 2 quorum size (was 3)

Once a leader is elected again, remove the dead peers with peer-remove as above. If more than one JetStream domain is present, name the domain explicitly with --js-domain <domain> — each domain runs its own meta group with its own quorum, and rescue only lowers one at a time.

Rescue is a last resort, not routine peer management. Lowering the quorum makes a smaller set of servers authoritative over the cluster’s metadata, which risks split-brain if a server you thought was permanently gone is actually still running on the other side of a partition. Only rescue when the missing servers are genuinely never coming back.

Short-term: stabilize the cluster

Wait for full meta group recovery before making changes. After restoring quorum, don’t immediately start creating streams or making configuration changes. Let the meta group fully synchronize:

# Monitor until all peers show Offline: false and Lag: 0
nats server report jetstream

If the outage was caused by rolling restarts, fix the procedure. Always verify meta group health between server restarts:

# Restart procedure: one server at a time
systemctl restart nats-server  # on server-1

# Wait for it to rejoin and catch up
watch -n 2 'nats server report jetstream 2>/dev/null | head -20'
# Only proceed to next server when Offline: false and Lag: 0

systemctl restart nats-server  # on server-2
# ... repeat

Long-term: prevent quorum loss

Use five-node clusters for critical workloads. A five-node meta group tolerates two simultaneous failures. This gives you room to lose one server unexpectedly and still have fault tolerance during a rolling restart of another:

1
// Go: verify cluster health before proceeding with operations
2
nc, _ := nats.Connect("nats://localhost:4222",
3
    nats.UserInfo("sys_user", "sys_pass"),
4
)
5
js, _ := nc.JetStream()
6

7
// Check JetStream availability before critical operations
8
info, err := js.AccountInfo()
9
if err != nil {
10
    log.Fatalf("JetStream unavailable (possible quorum loss): %v", err)
11
}
12
log.Printf("JetStream healthy: %d streams, %d consumers",
13
    info.Streams, info.Consumers)

1
# Python: health check before JetStream operations
2
import nats
3

4
async def ensure_js_available():
5
    nc = await nats.connect()
6
    js = nc.jetstream()
7

8
    try:
9
        info = await js.account_info()
10
        print(f"JetStream healthy: {info.streams} streams")
11
    except Exception as e:
12
        print(f"JetStream unavailable (possible quorum loss): {e}")
13
        raise

Distribute servers across failure domains. Never place two meta group peers on the same rack, hypervisor host, or availability zone. Use placement constraints in your orchestrator to ensure physical separation.

Implement automated health monitoring. Alert before quorum is lost — when the first replica goes offline (server-058), you have time to act.

Synadia Insights evaluates meta quorum health every collection epoch. It alerts on offline replicas (server-058) as an early warning and on quorum loss (system-007) as a critical event, giving operators the maximum time window to intervene.

Frequently asked questions

Do existing streams and consumers keep working without meta quorum?

Yes, with caveats. Streams whose own Raft groups have quorum continue accepting publishes and delivering messages. Consumers continue processing. But nothing can be changed — no new streams, consumers, or configuration updates. If any stream loses its own quorum during the meta outage, it can’t be recovered until meta quorum is restored. The cluster is operating on borrowed time.

How long can a cluster survive without meta quorum?

Indefinitely for existing, healthy workloads. But the risk compounds over time. Any stream that loses quorum can’t be repaired. Any consumer that needs reassignment is stuck. Any new workload can’t be deployed. The longer the outage, the more likely a secondary failure creates an unrecoverable situation.

Can I force a single remaining server to become the leader?

Not safely through normal operations. Raft requires a quorum by design — this is a safety guarantee, not a limitation. If you could force a single server to become leader, you’d risk split-brain if the other servers are actually still running on the other side of a partition. If servers are truly permanently lost and a leader can still be elected, remove them with nats server cluster peer-remove to reduce the group size and allow the remaining servers to form quorum. If no leader can be elected at all, peer-remove can’t commit either — use nats server cluster rescue (NATS 2.15+) to lower the quorum requirement first, then peer-remove the dead peers once a leader is back.

What’s the difference between meta quorum lost and stream quorum lost?

Meta quorum lost (system-007) affects the JetStream control plane — API operations, stream management, consumer assignment. Stream quorum lost (stream-007) affects a specific stream’s data plane — that stream can’t accept writes or elect a leader. Meta quorum loss is broader in scope but doesn’t necessarily affect data flow for healthy streams. Stream quorum loss is narrower but directly causes data unavailability for the affected stream.

How do I do rolling restarts without losing quorum?

Restart one server at a time. After each restart, run nats server report jetstream and wait until the restarted server shows Offline: false and Lag: 0 in the meta group table. Only then proceed to the next server. For a three-node cluster, this means you never have more than one server down simultaneously. For added safety, use lame duck mode (kill -SIGTERM <pid>) to gracefully drain the server before stopping it.