NATS Stream Leader Not Co-located

A stream leader is not co-located when the server holding the stream’s Raft leadership is in a different cluster than the one where most of the account’s client connections live. The stream works correctly, but every publish acknowledgment, every message delivery, and every ack crosses the inter-cluster gateway before it reaches the client that cares about it.

Why this matters

The stream leader is the server that owns the write path. A publish is routed to the leader, the leader replicates it to the other peers, and only the leader sends the acknowledgment back to the publisher. If the publisher is in cluster us-east and the leader is in eu-west, the publish crosses the gateway on the way in and the ack crosses it on the way back. The publisher sees the full round trip on every message it waits on.

The read path inherits the same cost, and it inherits it twice over, because consumers are not placed independently. A consumer’s Raft group is drawn from the servers that already hold its stream, so a consumer cannot have a leader in a cluster where the stream has no replica. When the stream’s leadership sits away from the clients, consumer leadership tends to sit there too: the fetch request crosses the gateway, the message crosses back, and the ack crosses again. For an AckExplicit consumer with a tight ack_wait, that added latency comes directly out of the time the application has to do its work, and in the worst case it pushes normal processing close enough to the deadline to trigger redeliveries.

This is why the finding is reported once per stream rather than once per consumer. A stream with forty consumers on the wrong side of a gateway has one placement problem, not forty, and the stream is the object whose placement an operator can actually change.

The check is an optimization signal, not an outage. It only fires when the connection picture is unambiguous: the account must have at least min_conns client connections in total, and a single cluster must hold at least min_dominance_percent of them. An account with three connections, or an account genuinely split evenly across two regions, is left alone, because there is no clearly better place to put the leader.

Common causes

• Raft elected a leader without regard for clients. Leader election optimizes for log currency and availability. The peer with the most up-to-date log wins, whatever cluster it sits in. Nothing in the election considers where the account’s connections are.

• The stream was created without placement. A stream created with no --cluster and no --tag is assigned by the meta leader using available resources. If the client fleet is in one cluster and spare capacity is in another, the stream lands in the wrong place and stays there.

• The stream was created from the wrong place. Streams created by a CI pipeline, a bootstrap job, or an operator connected to a management cluster inherit that vantage point. The workload that later uses the stream connects somewhere else.

• Clients moved and the stream did not. The application fleet was migrated to a new region or a new cluster was added closer to users. The stream continues to serve correctly from where it was first placed, so nothing fails and nothing prompts a review.

• A failover relocated leadership. A server restart, a rolling upgrade, or a manual step-down triggers a new election. If the stream’s replicas span clusters, the new leader can land on the far side and remain there indefinitely.

• Replicas were spread across clusters for durability. Placing R3 across three clusters survives losing a whole cluster, but it also means two thirds of the candidate leaders are remote from any given client population. Durability and locality pull in opposite directions here.

How to diagnose

Find the stream leader and its cluster

nats stream info <stream_name>

The cluster section names the leader server and lists the replicas. The Cluster line names the cluster the leader is reporting from. For a fleet-wide view:

nats stream report --leaders

This lists every stream with its leader, which makes an entire account’s placement visible in one pass.

Map servers to clusters

Leader names on their own are not enough if your server naming does not encode the cluster:

nats server list --sort cluster

The output pairs each server with its cluster and connection count, which is the lookup table for everything below.

Find where the account’s connections actually are

nats server report connections --account <account_name>

Connections are reported per server, so group by the cluster each server belongs to. To count one cluster at a time:

nats server report connections --account <account_name> --cluster us-east
nats server report connections --account <account_name> --cluster eu-west

This is the comparison the check makes. If one cluster holds a clear majority of the account’s client connections and the leader is somewhere else, the finding is correct.

Confirm a replica exists in the client cluster

A leader can only move to a server that already holds a replica:

nats stream info <stream_name> --json | jq '.cluster'

Cross-reference the replicas[].name values against nats server list. If none of the peers are in the cluster where the clients connect, stepping down will never help and the fix is a placement change rather than an election.

Check what the gateway is carrying

nats server report gateways

Sustained traffic on a gateway between the client cluster and the leader’s cluster is the cost this finding is describing. If several streams for the same account are misplaced in the same direction, that traffic is the aggregate of all of them, and it is also what triggers the account-scoped gateway checks.

How to fix it

Immediate: step down to a better-placed peer

If the stream already has a replica in the client’s cluster, ask the current leader to resign and name the peer you want to take over:

nats stream cluster step-down <stream_name> --preferred <server_name>

--preferred expresses a preference, not an assignment. Raft still runs an election and the outcome depends on peer state, so verify the result rather than assuming it:

nats stream info <stream_name> --json | jq '.cluster.leader'

Without --preferred, a plain step-down triggers an election with no hint at all and can hand leadership back to the same cluster:

nats stream cluster step-down <stream_name>

The stream is briefly unavailable during the election, typically well under a second. In-flight publishes that had not been acknowledged may need to be retried, and messages delivered but not yet acked will be redelivered by the new leader.

Short-term: change where the stream is placed

If the stream has no peer near the clients, leadership cannot move there and placement has to change. Pin the stream to the right cluster:

nats stream edit <stream_name> --cluster us-east

Editing placement on a clustered stream moves its assets: the new peers are assigned, the data is copied to them, and the old peers are released once the move completes. Plan for the copy. A large stream moves at whatever rate the link between the clusters allows, and the stream stays available throughout, still serving from its existing leader until the new peers catch up.

Tags are the better lever when you want placement expressed in terms of topology rather than a single named cluster:

nats stream edit <stream_name> --tag region:us-east

This requires servers to carry matching tags in their configuration:

1
server_name: us-east-1
2
cluster {
3
  name: us-east
4
}
5
server_tags: ["region:us-east", "az:us-east-1a"]

Once the stream holds peers in the target cluster, step down to move leadership there.

Create new streams with placement stated up front so this never becomes a migration:

nats stream add ORDERS --cluster us-east --replicas 3 --subjects 'orders.>'

Long-term: make placement a property of the workload

Decide placement from the connection map, not from spare capacity. For each stream, name the cluster its clients connect to and record that as the intended placement. A stream whose publishers and consumers are in one region has no reason to lead from another.

Revisit placement after topology changes. Adding a cluster, migrating an application fleet, or moving a customer to a new region invalidates placement decisions that were correct when they were made. Nothing breaks, so nothing raises its hand. Treat a client migration as including a stream placement review.

Re-check leadership after failovers. Rolling upgrades and server restarts move leadership and do not move it back. Re-running nats stream report --leaders after maintenance catches the drift while it is still cheap to correct.

Separate durability placement from locality. If a stream needs to survive the loss of a cluster, put the replicas where they survive it and accept that some clients are remote, or keep the stream local and use a mirror in the remote cluster for reads. A mirror serves remote readers from a local copy instead of paying a gateway hop per message.

Frequently asked questions

Why is this reported on the stream instead of on each consumer?

Because placement is a property of the stream. Consumer Raft groups are formed from the servers that already hold the stream, so a consumer’s leader can only be in a cluster where the stream has a replica. Reporting the same placement fact once per consumer produces dozens of findings for a single decision, and none of them can be acted on individually. Fixing the stream fixes the consumers.

The account’s connections are spread over several clusters. Is the finding still meaningful?

Only when one cluster is clearly dominant. The check requires the account to have at least min_conns connections and requires the leading cluster to hold at least min_dominance_percent of them before it says anything. Below either bar it stays silent, because there is no single better location to recommend. The finding reports the dominant cluster and its share so you can judge the margin yourself.

The check counts connections per account, but my stream is used by one service. Does that matter?

It can. The comparison uses the account’s overall client connection distribution as the proxy for where a stream’s clients are, because connections do not record which stream they use. For an account whose services all connect in the same place, that proxy is exact. For an account where one service connects in a different cluster than the rest, a stream used only by that service may be flagged even though its own clients are next to the leader. Check which service uses the stream before moving it.

How much latency does the gateway hop add?

Approximately one gateway round trip per operation that reaches the leader. Between clusters in the same region that is typically a few milliseconds. Across regions it is commonly 20 to 100 milliseconds or more. Multiply that by the operations involved: a synchronous publish pays it once, and a consume with an explicit ack pays it on the fetch, the delivery, and the ack.

Can I pin leadership to a specific server permanently?

No. Raft has no leader pinning, and --preferred influences a single election rather than establishing a standing preference. Any subsequent election can move leadership again. The durable control is placement: keep the stream’s peers in the cluster where the clients are, so that every candidate the election can choose is a good one.