Avoiding NATS Server Name Collisions in Leaf Node Hub-and-Spoke Topologies

A community member asked how to connect multiple Kubernetes NATS clusters to a hub using leaf nodes when each Kubernetes cluster uses the default pod name as its NATS server_name.

The short answer: any NATS servers that can see each other in a leaf node, route, or mesh topology need globally unique server names. If multiple Kubernetes clusters all contain servers named nats-server-0, nats-server-1, and nats-server-2, those names can collide when the clusters are joined through a hub.

Why the collision happens

In many Kubernetes deployments, the NATS server name is derived from the pod name. That is convenient inside a single cluster because StatefulSet pod names are stable, for example:

1
nats-server-0
2
nats-server-1
3
nats-server-2

However, if you deploy the same chart or manifest into multiple Kubernetes clusters, each cluster may have the same set of server names. When those NATS clusters connect to a common hub as leaf nodes, servers with the same names may become visible to the hub under the same account or topology.

NATS uses server names in several important places, including:

Identifying nodes in a cluster
Avoiding routing loops
JetStream cluster and metadata coordination
Operational reporting and cluster management

So a duplicate remote leaf node connection such as Remote: nats-server-1 is not just a cosmetic naming issue. It indicates that the topology is ambiguous to NATS.

The preferred fix: make server names globally unique

For new clusters, set unique NATS server names before the cluster is formed and before JetStream data is created.

A common pattern is to include a site, region, environment, or Kubernetes cluster identifier in the server name:

1
edge-eu1-nats-0
2
edge-eu1-nats-1
3
edge-eu1-nats-2
4

5
edge-us1-nats-0
6
edge-us1-nats-1
7
edge-us1-nats-2

The exact mechanism depends on how you deploy NATS, but the important property is this:

No two NATS servers that can see each other should have the same server_name.

This is especially important in hub-and-spoke leaf node topologies where multiple edge clusters connect to the same hub.

Also use unique cluster names and JetStream domains

Server names are not the only names that matter.

For multi-cluster topologies, also plan for:

Unique NATS cluster names across connected clusters
Unique JetStream domains when multiple JetStream-enabled clusters are connected or referenced

JetStream domains are particularly important when creating stream mirrors or sources across clusters. They let clients and servers address the correct JetStream API endpoint instead of relying on an ambiguous default.

For example, an edge cluster might use a domain such as:

1
EDGE_EU1

Another edge cluster might use:

1
EDGE_US1

The hub might use:

1
HUB

The names themselves are up to you. The key is that they are unique and consistently used in your stream source or mirror configuration.

Do not casually rename a live JetStream server

If a NATS cluster is already live and has JetStream data, changing server_name in place can break the cluster.

That is expected behavior. From the perspective of the JetStream cluster, a renamed server can look like:

One known peer disappeared.
A different peer appeared.
The cluster now has stale metadata for the old peer name.
Replicas may not repair until the old peer is explicitly removed.

For this reason, do not treat server_name as a normal mutable setting in a live JetStream cluster.

Safer migration options for existing clusters

If the clusters are already live, there is no single one-step rename process. Use an operational migration process and test it before applying it to production.

Option 1: Grow with correctly named servers, then remove old peers

The safer pattern is:

Temporarily grow the cluster by adding new NATS servers with globally unique names.
Wait for the cluster to become healthy.
Remove the old peers one at a time.
Wait for JetStream to repair and rebalance before removing the next old peer.

The relevant CLI operation is:

nats server cluster peer-remove <server>

If the server name is not clear, inspect JetStream server state first:

nats server report jetstream

Then remove by the appropriate server name or server ID:

nats server cluster peer-remove <server-or-id>

These are advanced cluster management operations and typically require access through the system account. Use them carefully.

Option 2: Rename one server at a time, then remove the stale peer

In constrained environments where temporarily growing the cluster is difficult, another possible approach is to change one server at a time and then remove the stale peer entry for the old name.

Conceptually, that process is:

Stop or rename one server.
Let it rejoin with the new globally unique name.
Confirm cluster health.
Remove the old peer name using peer-remove.
Repeat for the next server.

This is more delicate than adding new capacity first. If you are running with strict anti-affinity, limited node capacity, or production JetStream data, plan a maintenance window and validate the process in a representative environment.

Can accounts isolate clusters enough to avoid the collision?

Sometimes, yes, but treat this as an isolation design choice rather than a replacement for good naming.

If two leaf clusters connect to different accounts on the hub, they may not be aware of each other in the same way. For example:

1
Leaf cluster 1 -> hub account APP_TENANT1
2
Leaf cluster 2 -> hub account APP_TENANT2

That can be a valid multi-tenancy pattern when you want the leaf clusters to have no awareness of each other.

A few important details:

The account names on the leaf side do not have to match the account names on the hub side.
The user or JWT used by the leaf node connection determines which hub account the leaf connects into.
Separate hub accounts can reduce visibility between tenants.
If servers with duplicate names later become visible to each other through topology changes, imports, exports, or account design, duplicate names can still become a problem.

So accounts can help with tenant isolation, but the robust rule remains: use globally unique server names for every server that may become visible in the mesh.

What about a hub account that can mirror streams from all leaves?

A common hub-and-spoke design is to keep leaf clusters isolated in tenant accounts, then allow a central account on the hub to mirror or source selected streams.

At a high level, that requires:

Unique JetStream domains for the leaf clusters.
Unique stream names where they will be referenced centrally.
Account exports from the tenant accounts.
Account imports into the central account.
Mirror or source configuration that points at the correct domain and subject permissions.

The exact import and export subjects depend on your stream names, domains, and permissions model. JetStream stream source and mirror behavior is documented in the NATS protocol reference:

https://docs.nats.io/reference/reference-protocols/nats_api_reference#stream-source-and-mirror

When centralizing streams from multiple leaves, avoid duplicate stream names unless you have a deliberate naming and permissions strategy. If two leaves both expose a stream called ORDERS into the same central context, you are creating ambiguity for operators and configuration, even if the underlying accounts are separate.

A safer convention is to include the tenant, site, or domain in stream names when they will be referenced by a shared hub account:

1
TENANT1_ORDERS
2
TENANT2_ORDERS
3
EDGE_EU1_ORDERS
4
EDGE_US1_ORDERS

Can this be automated?

Yes, but automation should use the same NATS APIs and operational sequence that the CLI uses.

A practical way to understand what the CLI is doing is to run a nats command with tracing enabled. Tracing prints the NATS subjects and request/response flow behind the command:

nats --trace server report jetstream

Note that --trace only adds visibility. It does not turn a command into a dry run: a traced nats server cluster peer-remove still performs the removal. To inspect the request flow for destructive operations safely, trace them against a non-production cluster first.

The trace output shows the NATS subjects and request flow used by the command, which can help when implementing operational tooling in Go or another language.

Do not automate peer removal or JetStream cluster changes until you have tested failure cases, retries, and health checks. These operations affect cluster metadata and stream replica repair.

Practical checklist

Before connecting multiple Kubernetes NATS clusters through a hub:

Give every NATS server a globally unique server_name.
Give every NATS cluster a unique cluster name.
Use unique JetStream domains for each JetStream-enabled cluster.
Decide whether each leaf should connect to the same hub account or separate tenant accounts.
If using central mirroring, design stream names, exports, imports, and permissions explicitly.
Test the configuration locally or in a non-production environment before applying it through Helm or GitOps.
For live JetStream clusters, do not rename all servers in place. Use a planned migration process.

Summary

In a NATS hub-and-spoke topology, duplicate Kubernetes pod names can become duplicate NATS server names. That is safe only while those servers are isolated. Once clusters are connected through leaf nodes and can see each other, server names must be globally unique.

For new deployments, fix this up front with unique server names, cluster names, and JetStream domains. For existing JetStream clusters, avoid direct renames; add correctly named peers and remove old peers carefully, or rename one node at a time with explicit peer cleanup. Accounts can provide tenant isolation, but they are not a universal substitute for unambiguous topology naming.

Want help from the NATS experts? Meet with our architects to get help tailored to your use case and environment.