NATS Uneven Leader Distribution: What It Means and How to Fix It

Uneven leader distribution means one or more servers in a JetStream cluster are hosting significantly more Raft group leaders than their fair share, concentrating write load and increasing the risk of resource exhaustion on those servers.

Why this matters

In a JetStream cluster, every replicated stream and durable consumer operates a Raft consensus group. The leader of each group handles all write operations — publishing messages, acknowledging deliveries, updating consumer state. Followers receive replicated data but don’t handle client requests directly. This means the leader bears a disproportionate share of the CPU, I/O, and network load for its Raft group.

When leaders are evenly distributed across cluster nodes, the write load is balanced. In a three-node cluster with 30 stream leaders, each server ideally hosts about 10 leaders. If one server ends up with 20 leaders while the others have 5 each, that server handles four times the write load of its peers. It uses more CPU for Raft proposals, more disk I/O for writes, and more network bandwidth for replication. The other servers sit relatively idle — they have capacity that isn’t being used.

The imbalance tends to be self-reinforcing. The overloaded server becomes slower, which increases write latency for all streams it leads. If the slowdown becomes severe enough, Raft groups may time out and trigger leader elections — but those elections may elect leaders on the same server if it recovers before the election completes. In the worst case, the overloaded server becomes resource-constrained enough to affect route and client connections, cascading into broader cluster instability. What starts as a performance optimization issue can escalate into an operational incident.

Common causes

Server restart or recovery. When a server goes down, all its Raft groups elect new leaders on the remaining nodes. When the server comes back, it rejoins as a follower — leaders don’t automatically migrate back. The surviving nodes retain the extra leaders indefinitely until explicitly rebalanced.
Rolling cluster upgrades. Each server restart during a rolling upgrade shifts leaders to other nodes. After the upgrade completes, leaders are concentrated on the servers that restarted last (or didn’t restart at all).
Asymmetric hardware. Servers with faster disks or lower network latency tend to win Raft elections more often. Over time, leaders naturally accumulate on the “fastest” node even without explicit failures.
Manual step-downs without redistribution. Operators stepping down leaders for maintenance concentrate them on specific nodes. Without a follow-up redistribution, the imbalance persists.
Cluster scaling. Adding a new server to an existing cluster doesn’t automatically redistribute leaders. The new server starts with zero leaders and only gains them through explicit step-downs or future elections.

How to diagnose

Check leader distribution per server

# JetStream report including the RAFT leader distribution
nats server report jetstream --leaders

The --leaders flag appends a RAFT Leader Report showing how many stream and consumer leaders each server hosts, along with each server’s share of the total. (In the summary table above it, the * next to a server name marks the meta-group leader only — it says nothing about stream or consumer leaders.) Compare leader counts across servers. Any server with more than 1.5x the average is a hotspot.

List which streams are led by which server

# Stream report with cluster leader information
nats stream report

The Replicas column lists each stream’s peers, with * marking the current leader. This tells you exactly which streams to step down for rebalancing. (nats stream report --leaders adds the same per-server leader summary as the server report.)

Measure the performance impact

# Compare traffic per server
nats server report connections --sort in-msgs

Compare message rates and connection counts across servers (for CPU, use nats server report cpu). The leader-heavy server will typically show higher values in throughput metrics.

Compare against expected distribution

For a quick check, divide total leaders by server count to get the expected average, then flag servers exceeding 1.5x that number. For scripting, request the raw JSz data — each server replies with one JSON document, and the --raft flag is required for the streams_leader and consumers_leader counts to be populated (they’re omitted entirely when zero, hence the // 0 defaults):

# Raw JSz data for scripting
nats server request jetstream --raft | jq -s 'map({name: .server.name, stream_leaders: (.data.streams_leader // 0), consumer_leaders: (.data.consumers_leader // 0)}) | sort_by(-.stream_leaders)'

How to fix it

Immediate: rebalance with the NATS CLI

The NATS CLI ships convenience commands that find the leaders in each cluster, compute an even distribution, and step down leaders from over-represented servers — using placement hints to steer leadership toward the under-loaded ones — until the distribution evens out. This is the fastest way to correct an imbalance and is the officially supported mechanism — the server itself will not do this for you (see the FAQ below):

# Rebalance stream leaders across the cluster
nats stream cluster balance

# Rebalance the consumer leaders of a specific stream
nats consumer cluster balance <stream_name>

The balancer requires NATS Server 2.11.0 or newer, which added support for placement hints in step-down requests. (Don’t confuse these with nats server cluster balance — that command rebalances client connections across servers, not leaders.) Re-run nats server report jetstream --leaders afterwards to confirm the new distribution.

Targeted: step down individual leaders

When you need finer control — for example, moving a specific hot stream off a specific server — use nats stream cluster step-down and nats consumer cluster step-down to trigger new leader elections. The current leader becomes a follower, and one of the other servers is elected:

# Step down the leader for a specific stream
nats stream cluster step-down <stream_name>

# Step down a consumer leader
nats consumer cluster step-down <stream_name> <consumer_name>

Target the streams and consumers led by the overloaded server until the distribution approaches the cluster average. Start with the servers that have the highest leader counts for maximum impact. Check the distribution after each batch:

# Verify the new distribution
nats server report jetstream --leaders

Short-term: script a custom rebalance

nats stream cluster balance and nats consumer cluster balance cover the common case. Script your own logic when you need placement rules the CLI doesn’t know about — for example, keeping certain streams pinned to specific servers, weighting by throughput rather than raw leader count, or coordinating with a maintenance window. The server can’t infer those rules on your behalf, so the balancing decision belongs in your tooling:

1
package main
2

3
import (
4
  "context"
5
  "fmt"
6

7
  "github.com/nats-io/nats.go"
8
  "github.com/nats-io/nats.go/jetstream"
9
)
10

11
func main() {
12
  nc, _ := nats.Connect(nats.DefaultURL)
13
  js, _ := jetstream.New(nc)
14

15
  ctx := context.Background()
16

17
  // Count leaders per server
18
  leaderCount := make(map[string]int)
19
  var streams []string
20

21
  sl := js.ListStreams(ctx)
22
  for si := range sl.Info() {
23
    if si.Cluster != nil && si.Cluster.Leader != "" {
24
      leaderCount[si.Cluster.Leader]++
25
      streams = append(streams, si.Config.Name)
26
    }
27
  }
28

29
  total := 0
30
  for _, count := range leaderCount {
31
    total += count
32
  }
33
  avg := total / len(leaderCount)
34
  threshold := avg + avg/2 // 1.5x average
35

36
  for server, count := range leaderCount {
37
    if count > threshold {
38
      excess := count - avg
39
      fmt.Printf("Server %s has %d leaders (avg %d), stepping down %d\n",
40
        server, count, avg, excess)
41
      // Step down excess leaders from this server
42
    }
43
  }
44
}

1
import json
2
import subprocess
3

4
def rebalance_leaders():
5
    # Get current leader distribution; each server replies
6
    # with one JSz JSON document on its own line
7
    result = subprocess.run(
8
        ["nats", "server", "request", "jetstream", "--raft"],
9
        capture_output=True, text=True
10
    )
11

12
    responses = [json.loads(line) for line in result.stdout.splitlines() if line.strip()]
13

14
    leader_counts = {
15
        r["server"]["name"]: r["data"].get("streams_leader", 0) for r in responses
16
    }
17
    avg = sum(leader_counts.values()) // len(leader_counts)
18
    threshold = int(avg * 1.5)
19

20
    for server, count in leader_counts.items():
21
        if count > threshold:
22
            excess = count - avg
23
            print(f"{server}: {count} leaders (avg {avg}), need to step down {excess}")
24
            # Step down streams led by this server

Long-term: bake rebalancing into your operations

nats-server does not — and by design will not — rebalance leaders on its own. An automatic rebalance would move leadership out from under active workloads at times the server can’t reason about, so the project keeps the decision in the operator’s hands and exposes the tooling (nats stream cluster balance, nats consumer cluster balance, and the individual step-down commands) instead. That means rebalancing has to live in your runbook or your automation.

Rebalance after every maintenance event. Make leader redistribution a standard step in your runbook for server restarts, upgrades, and scaling operations. A post-maintenance nats stream cluster balance (or scripted step-down sweep) prevents imbalance from accumulating.

Automate on your schedule, not the server’s. Wire the balance commands — or your custom script — into CI/CD, a cron job, or your incident-response tooling so rebalancing happens on a cadence and in windows you control.

Monitor leader distribution continuously. Synadia Insights evaluates leader distribution automatically and flags servers exceeding the 1.5x threshold. Catching imbalance early — before it impacts performance — avoids the cascading effects of a severely overloaded node.

Frequently asked questions

Will stepping down a stream leader cause message loss?

No. Leader step-down triggers a clean Raft leader election. The new leader takes over from the committed state of the Raft log. No messages are lost, and the transition takes milliseconds. Clients may see a brief delay (typically under 100ms) while the new leader is elected and begins accepting writes. For JetStream consumers, delivery continues transparently.

Does NATS automatically rebalance leaders?

No. nats-server never rebalances leaders on its own, and automatic server-side rebalancing is not planned. Moving leadership is disruptive — it briefly pauses writes for the affected Raft groups — and the server has no way to know when that disruption is acceptable for your workload or which placement is “correct” for your topology. Instead, the server exposes the step-down primitives and the NATS CLI ships convenience commands (nats stream cluster balance and nats consumer cluster balance, requiring NATS Server 2.11.0 or newer) built on them. You run them — from a script, a runbook step, or a scheduled job — when the timing suits you. For anything more opinionated (pinning specific streams to specific servers, weighting by throughput, coordinating with other maintenance), script the rebalance yourself using the same step-down primitives.

How many leaders per server is too many?

There’s no absolute number — it depends on your hardware, stream throughput, and message sizes. The check uses a relative threshold: a server hosting more than 1.5x the cluster average number of leaders is flagged. In a three-node cluster with 90 total Raft groups, the average is 30 per node; a server with 46+ leaders would trigger the check. The performance impact depends on how active those streams are — 50 leaders for mostly-idle streams may be less impactful than 35 leaders for high-throughput streams.

Should I also rebalance consumer leaders?

Yes. Consumer Raft groups have the same leader/follower dynamics as stream groups. Consumer leaders handle acknowledgement processing and delivery state updates. Use nats consumer cluster step-down to rebalance consumer leaders alongside stream leaders for complete balance.

Does adding a new server to the cluster automatically fix the distribution?

No. A new server starts with zero leaders. Existing Raft groups don’t spontaneously elect the new server as leader — they only do so during an election triggered by the current leader stepping down or becoming unavailable. After adding a new server, explicitly step down leaders to redistribute across the expanded cluster.

FEATURED

RESOURCES

Comparisons