NATS Leader Disagreement: What It Means and How to Fix It

Multiple servers in the meta cluster report themselves as the leader. In a healthy Raft group, exactly one leader exists at any time. This condition typically indicates a transient state during a leader election rather than a true split-brain — Raft quorum rules prevent split-brain in correctly configured clusters. Multiple servers may briefly report leadership during term transitions. If persistent, it points to a network partition or stale state that needs investigation.

Why this matters

The meta cluster leader is the single point of coordination for all JetStream operations. It handles stream creation and deletion, consumer assignments, placement decisions, and configuration changes. Raft consensus guarantees that only one leader exists and that all decisions are agreed upon by a majority of peers before being committed. When two servers both believe they are the leader, that guarantee is broken.

In a true split-brain scenario — where a network partition divides the cluster into two groups, each electing its own leader — conflicting operations can proceed on both sides. One partition might create a stream while the other deletes it. One might assign a consumer to a server that the other partition considers offline. When the partition heals and the groups merge, the inconsistent state must be reconciled. Raft’s design handles this by having the leader with the higher term “win,” but the operations committed by the losing side are rolled back, which can mean lost JetStream API operations that clients believed succeeded.

Even when the disagreement is transient — a server briefly reports stale leader information after rejoining from a partition — it indicates a fragile network or cluster configuration. Operators and monitoring systems querying different servers may get contradictory answers about cluster state. Automation that reacts to leader information (placement decisions, health checks, routing) can make incorrect decisions based on stale data. Leader disagreement is a signal that the cluster’s consensus layer is or was recently compromised.

Common causes

• Network partition. The classic split-brain trigger. A network failure divides the cluster into two groups. If each group has enough members to form a quorum (possible with even-numbered clusters or complex multi-partition scenarios), each group elects its own leader. This is the most dangerous scenario because both leaders can accept writes.

• Stale leader state after partition recovery. A server was partitioned, fell behind on Raft state, and when it rejoins, it briefly reports the old leader from its last known state. The disagreement is transient — once the server catches up on Raft log entries, it learns the current leader. But monitoring systems may detect the disagreement window.

• Overloaded meta leader. The current leader is under heavy load (high JetStream API rate, disk I/O contention, CPU saturation) and fails to send Raft heartbeats in time. Followers trigger an election and elect a new leader, but the old leader hasn’t yet realized it’s been deposed. Both briefly report as leader until the old leader receives a message with a higher term and steps down.

• Even-numbered cluster size. A cluster with an even number of JetStream peers (e.g., 4 nodes) can split into two equal groups during a partition. Neither group has a strict majority, but depending on timing, each group may attempt an election. This is why NATS production clusters should always have an odd number of peers.

• Clock skew or extreme latency. Raft relies on timeouts to detect leader failure. If clock drift or extreme network latency causes followers to misinterpret heartbeat timing, they may trigger unnecessary elections. With servers on different sides of a high-latency link, the risk of concurrent elections increases.

How to diagnose

Check which servers claim to be leader

Query the meta group status:

nats server report jetstream

The Raft Meta Group table shows a Leader column. In a healthy cluster, exactly one server shows Leader: yes. If multiple servers show Leader: yes, or if different servers report different leaders, disagreement is confirmed.

Query each server’s view independently

To see what each individual server believes about the meta leader:

# Query all servers for their JetStream state
nats server req jetstream

This returns each server’s view of the meta cluster, including which server it considers the leader. Compare responses — if servers report different leaders, you have active disagreement.

Check for network partition indicators

# Check route connectivity between all cluster members
nats server list

Servers that can’t see each other won’t appear in each other’s server list output. If you can access monitoring ports on both sides:

# From server A
curl -s http://<server-a>:8222/routez | jq '.routes[].remote_id'

# From server B
curl -s http://<server-b>:8222/routez | jq '.routes[].remote_id'

Missing routes between servers indicate a network partition.

Check server logs for election activity

1
[INF] JetStream cluster new meta leader
2
[WRN] Raft election timeout, starting new election
3
[INF] Raft elected leader for meta group, term <N>

Multiple election messages in a short window, or elections with different terms on different servers, confirm the disagreement. The term number is critical — the highest term wins in Raft.

Verify clock synchronization

Extreme clock drift can cause Raft timing issues:

# Check NTP status on each server
timedatectl status
# or
ntpstat

Clock differences of more than a few hundred milliseconds between cluster peers are a risk factor.

How to fix it

Immediate: determine if this is transient or persistent

Understand the nature of the disagreement. This condition most commonly indicates a transient state during a leader election, not a split-brain. Raft quorum rules prevent true split-brain — only one side of a partition can have a majority and elect a leader. Multiple servers may briefly report leadership during Raft term transitions as the old leader hasn’t yet received a message with the higher term.

If persistent, check for asymmetric partitions. A persistent disagreement (lasting more than a few seconds) suggests an asymmetric network partition where servers can partially communicate. Check network connectivity between all cluster peers.

Restart the lower-term leader to force convergence. If two servers claim leadership, compare their Raft terms (visible in server logs or via nats server req jetstream). The server with the lower term is the stale leader — restart it to force it to rejoin and accept the higher-term leader:

systemctl restart nats-server  # on the lower-term leader

Alternatively, step down the meta leader. This forces a clean election where all reachable peers participate:

nats server cluster step-down

This tells the current leader (whichever server the connected client considers the leader) to step down. A new election occurs immediately. If the network partition has healed, the election resolves the disagreement by establishing a single leader with a new, higher term.

If the partition is still active, fix the network first. Forcing a step-down while servers are still partitioned won’t resolve anything — each side will elect its own leader again. Restore network connectivity between all cluster peers before attempting a step-down.

Short-term: resolve the underlying partition

Restore network connectivity. Identify what’s blocking communication between servers:

# Test route port connectivity
nc -zv <remote-server> 6222

# Test gateway port if applicable
nc -zv <remote-server> 7522

# Check for firewall rules
iptables -L -n | grep <port>

Once connectivity is restored, NATS servers automatically re-establish routes. The Raft protocol detects the conflicting leaders and the one with the higher term (or the one that wins a new election) becomes the sole leader.

If servers have stale state, restart them. A server that was partitioned for an extended period may have accumulated stale Raft state:

systemctl restart nats-server

On restart, the server joins the meta group fresh, receives a snapshot from the current leader, and aligns its state. This is a safe operation — the restarting server will never become leader until it’s fully caught up.

Verify the resolution:

nats server report jetstream

Confirm exactly one server shows Leader: yes and all peers show Offline: false.

Long-term: prevent leader disagreement

Use odd-numbered cluster sizes. A three-node cluster cannot split into two groups that both have quorum. A five-node cluster can tolerate a two-node partition without split-brain. Never run production meta clusters with an even number of peers:

1
# Correct: 3 or 5 JetStream-enabled servers per cluster
2
jetstream {
3
  store_dir: /data/jetstream
4
}

Ensure reliable network between cluster peers. Cluster peers should be on the same network segment with low, consistent latency. For cross-datacenter clusters, use dedicated links or VPN tunnels with bandwidth guarantees. Raft is sensitive to latency variance — a network that’s usually 1ms but occasionally spikes to 500ms will cause election instability.

Monitor for leader disagreement proactively:

1
// Go: detect leader disagreement
2
nc, _ := nats.Connect("nats://localhost:4222",
3
    nats.UserInfo("sys_user", "sys_pass"),
4
)
5

6
// Collect meta state from all servers
7
inbox := nc.NewRespInbox()
8
sub, _ := nc.SubscribeSync(inbox)
9
nc.PublishRequest("$SYS.REQ.SERVER.PING.JSZ", inbox, nil)
10

11
leaders := map[string]bool{}
12
for {
13
    msg, err := sub.NextMsg(2 * time.Second)
14
    if err != nil {
15
        break
16
    }
17
    // Parse JSZ response, extract meta_leader field
18
    // Add to leaders map
19
}
20
if len(leaders) > 1 {
21
    log.Printf("ALERT: Meta leader disagreement detected: %v", leaders)
22
}

1
# Python: check leader agreement
2
import nats
3

4
async def check_leader_agreement():
5
    nc = await nats.connect(user="sys_user", password="sys_pass")
6

7
    responses = []
8
    inbox = nc.new_inbox()
9
    sub = await nc.subscribe(inbox)
10
    await nc.publish_request("$SYS.REQ.SERVER.PING.JSZ", inbox, b"")
11

12
    try:
13
        while True:
14
            msg = await sub.next_msg(timeout=2)
15
            responses.append(msg)
16
    except Exception:
17
        pass
18

19
    # Parse each response's meta_leader field
20
    # Alert if more than one unique leader reported

Synadia Insights detects leader disagreement automatically by comparing the reported meta leader across all servers every collection epoch. A critical alert fires immediately when disagreement is detected.

Frequently asked questions

Is leader disagreement always a split-brain?

No. The most common case is transient: a server rejoins after a brief partition and reports stale leader information for a few seconds until it catches up on Raft state. True split-brain — where two leaders are actively accepting writes — requires a network partition where both sides have quorum. This is rare in correctly configured (odd-numbered) clusters but extremely dangerous when it occurs.

What happens to JetStream operations during a leader disagreement?

It depends on which leader a client connects to. If a client’s server considers leader A the meta leader, API requests route to leader A. If another client’s server considers leader B the leader, its requests route to leader B. Both may succeed, but the operations may conflict. When the disagreement resolves, the leader with the lower Raft term has its operations rolled back.

How do I know if data was lost during a split-brain?

After the disagreement resolves, check the JetStream API for any operations that should have succeeded but didn’t persist. Streams that were created during the losing partition’s term will not exist. Consumer assignments from the losing side will be rolled back. Server logs on the losing side will show Raft entries being discarded as it catches up to the winning leader’s state.

Can I prevent split-brain entirely?

With an odd-numbered cluster, Raft’s design makes true split-brain impossible under simple partition scenarios — only one side can have a majority. However, complex network failures (asymmetric partitions, partial connectivity) can create edge cases. The best defense is reliable networking between peers, odd cluster sizes, and monitoring that detects disagreement immediately so you can intervene before both sides accumulate conflicting state.

Should I restart all servers to fix a leader disagreement?

No — that risks losing quorum entirely. Force a step-down with nats server cluster step-down first. If that doesn’t resolve it (because the partition is still active), fix the network. Only restart individual servers if they have corrupted or stuck Raft state, and always restart one at a time to maintain quorum.