NATS Uneven Leader Distribution: What It Means and How to Fix It

Uneven leader distribution means one or more servers in a JetStream cluster are hosting significantly more Raft group leaders than their fair share, concentrating write load and increasing the risk of resource exhaustion on those servers.

Why this matters

In a JetStream cluster, every replicated stream and durable consumer operates a Raft consensus group. The leader of each group handles all write operations — publishing messages, acknowledging deliveries, updating consumer state. Followers receive replicated data but don’t handle client requests directly. This means the leader bears a disproportionate share of the CPU, I/O, and network load for its Raft group.

When leaders are evenly distributed across cluster nodes, the write load is balanced. In a three-node cluster with 30 stream leaders, each server ideally hosts about 10 leaders. If one server ends up with 20 leaders while the others have 5 each, that server handles four times the write load of its peers. It uses more CPU for Raft proposals, more disk I/O for writes, and more network bandwidth for replication. The other servers sit relatively idle — they have capacity that isn’t being used.

The imbalance tends to be self-reinforcing. The overloaded server becomes slower, which increases write latency for all streams it leads. If the slowdown becomes severe enough, Raft groups may time out and trigger leader elections — but those elections may elect leaders on the same server if it recovers before the election completes. In the worst case, the overloaded server becomes resource-constrained enough to affect route and client connections, cascading into broader cluster instability. What starts as a performance optimization issue can escalate into an operational incident.

Common causes

Server restart or recovery. When a server goes down, all its Raft groups elect new leaders on the remaining nodes. When the server comes back, it rejoins as a follower — leaders don’t automatically migrate back. The surviving nodes retain the extra leaders indefinitely until explicitly rebalanced.
Rolling cluster upgrades. Each server restart during a rolling upgrade shifts leaders to other nodes. After the upgrade completes, leaders are concentrated on the servers that restarted last (or didn’t restart at all).
Asymmetric hardware. Servers with faster disks or lower network latency tend to win Raft elections more often. Over time, leaders naturally accumulate on the “fastest” node even without explicit failures.
Manual step-downs without redistribution. Operators stepping down leaders for maintenance concentrate them on specific nodes. Without a follow-up redistribution, the imbalance persists.
Cluster scaling. Adding a new server to an existing cluster doesn’t automatically redistribute leaders. The new server starts with zero leaders and only gains them through explicit step-downs or future elections.

How to diagnose

Check leader distribution per server

# JetStream report showing leader counts per server
nats server report jetstream

Look at the Leader column (or Streams and Consumers counts with leader indicators). Compare leader counts across servers. Any server with more than 1.5x the average is a hotspot.

List which streams are led by which server

# Stream report with cluster leader information
nats stream report

The cluster column shows which server leads each stream (marked with *). This tells you exactly which streams to step down for rebalancing.

Measure the performance impact

# Check CPU and I/O metrics per server
nats server report connections --sort in-msgs

Compare message rates, CPU usage, and connection counts across servers. The leader-heavy server will typically show higher values in throughput metrics.

Compare against expected distribution

For a quick check, divide total leaders by server count to get the expected average, then flag servers exceeding 1.5x that number:

# JSON output for scripting
nats server report jetstream --json | jq '[.[] | {name: .name, streams: .streams, leaders: .leader_count}] | sort_by(-.leaders)'

How to fix it

Immediate: step down leaders from the overloaded server

Use nats stream cluster step-down and nats consumer cluster step-down to trigger new leader elections. Target servers with the highest leader counts first. The overloaded server will become a follower, and one of the other servers will be elected leader:

# Step down the leader for a specific stream
nats stream cluster step-down <stream_name>

# Step down a consumer leader
nats consumer cluster step-down <stream_name> <consumer_name>

Target the streams and consumers led by the overloaded server until the distribution approaches the cluster average. Start with the servers that have the highest leader counts for maximum impact. Check the distribution after each batch:

# Verify the new distribution
nats server report jetstream

Short-term: script a balanced redistribution

For clusters with many Raft groups, manual step-downs are tedious. Script the rebalancing:

1
package main
2

3
import (
4
  "context"
5
  "fmt"
6
  "os/exec"
7
  "strings"
8

9
  "github.com/nats-io/nats.go"
10
  "github.com/nats-io/nats.go/jetstream"
11
)
12

13
func main() {
14
  nc, _ := nats.Connect(nats.DefaultURL)
15
  js, _ := jetstream.New(nc)
16

17
  ctx := context.Background()
18

19
  // Count leaders per server
20
  leaderCount := make(map[string]int)
21
  var streams []string
22

23
  sl := js.ListStreams(ctx)
24
  for si := range sl.Info() {
25
    if si.Cluster != nil && si.Cluster.Leader != "" {
26
      leaderCount[si.Cluster.Leader]++
27
      streams = append(streams, si.Config.Name)
28
    }
29
  }
30

31
  total := 0
32
  for _, count := range leaderCount {
33
    total += count
34
  }
35
  avg := total / len(leaderCount)
36
  threshold := avg + avg/2 // 1.5x average
37

38
  for server, count := range leaderCount {
39
    if count > threshold {
40
      excess := count - avg
41
      fmt.Printf("Server %s has %d leaders (avg %d), stepping down %d\n",
42
        server, count, avg, excess)
43
      // Step down excess leaders from this server
44
    }
45
  }
46
}

1
import asyncio
2
import subprocess
3

4
async def rebalance_leaders():
5
    # Get current leader distribution
6
    result = subprocess.run(
7
        ["nats", "server", "report", "jetstream", "--json"],
8
        capture_output=True, text=True
9
    )
10

11
    import json
12
    servers = json.loads(result.stdout)
13

14
    leader_counts = {s["name"]: s.get("leader_count", 0) for s in servers}
15
    avg = sum(leader_counts.values()) // len(leader_counts)
16
    threshold = int(avg * 1.5)
17

18
    for server, count in leader_counts.items():
19
        if count > threshold:
20
            excess = count - avg
21
            print(f"{server}: {count} leaders (avg {avg}), need to step down {excess}")
22
            # Step down streams led by this server

Long-term: automate leader rebalancing

Enable server-side leader rebalancing. Recent versions of nats-server include built-in leader rebalancing that automatically redistributes leaders after topology changes. Ensure your server version supports this feature and that it’s enabled.

Rebalance after every maintenance event. Make leader redistribution a standard step in your runbook for server restarts, upgrades, and scaling operations. A post-maintenance nats stream cluster step-down sweep prevents imbalance from accumulating.

Monitor leader distribution continuously. Synadia Insights evaluates leader distribution automatically and flags servers exceeding the 1.5x threshold. Catching imbalance early — before it impacts performance — avoids the cascading effects of a severely overloaded node.

Frequently asked questions

Will stepping down a stream leader cause message loss?

No. Leader step-down triggers a clean Raft leader election. The new leader takes over from the committed state of the Raft log. No messages are lost, and the transition takes milliseconds. Clients may see a brief delay (typically under 100ms) while the new leader is elected and begins accepting writes. For JetStream consumers, delivery continues transparently.

Does NATS automatically rebalance leaders?

Recent versions of nats-server (2.10+) include automatic leader rebalancing capabilities. However, the behavior varies by version and configuration. In older versions, leaders stay where elections placed them until explicitly stepped down. Even with automatic rebalancing enabled, the process is gradual — manual step-downs are faster for addressing acute imbalance after a node recovery.

How many leaders per server is too many?

There’s no absolute number — it depends on your hardware, stream throughput, and message sizes. The check uses a relative threshold: a server hosting more than 1.5x the cluster average number of leaders is flagged. In a three-node cluster with 90 total Raft groups, the average is 30 per node; a server with 46+ leaders would trigger the check. The performance impact depends on how active those streams are — 50 leaders for mostly-idle streams may be less impactful than 35 leaders for high-throughput streams.

Should I also rebalance consumer leaders?

Yes. Consumer Raft groups have the same leader/follower dynamics as stream groups. Consumer leaders handle acknowledgement processing and delivery state updates. Use nats consumer cluster step-down to rebalance consumer leaders alongside stream leaders for complete balance.

Does adding a new server to the cluster automatically fix the distribution?

No. A new server starts with zero leaders. Existing Raft groups don’t spontaneously elect the new server as leader — they only do so during an election triggered by the current leader stepping down or becoming unavailable. After adding a new server, explicitly step down leaders to redistribute across the expanded cluster.

FEATURED

RESOURCES

Comparisons