Checks/CHANGE_002

NATS JetStream Domain Changed: What It Means and How to Fix It

Severity
Warning
Category
Change
Applies to
Change
Check ID
CHANGE_002
Detection threshold
JetStream domain value changed between consecutive collection epochs

A JetStream domain change means a NATS server’s configured domain value in the jetstream block changed between two consecutive monitoring epochs. The domain is the namespace that scopes JetStream API access across leafnode topologies — changing it silently breaks every client, mirror, and source that references the old domain name.

Why this matters

The JetStream domain is not just a label. It defines the API prefix that clients use to reach JetStream from remote leafnode connections. When a server’s domain is set to hub, the JetStream API is accessible at $JS.hub.API.> from any leafnode connected to that server. Every client application, every stream mirror, and every stream source that crosses a leafnode boundary encodes this domain into its configuration. Change the domain, and all of those references point to an API prefix that no longer exists.

The failure mode is silent. Clients sending requests to the old domain prefix get no response — there’s no error, no rejection, just a timeout. Request-reply patterns hang. Mirrors and sources stop receiving data without raising an error, because the source subject they’re listening on no longer matches anything. In a multi-cluster leafnode topology, a single domain change can cascade into stalled data pipelines across your entire deployment before anyone notices the root cause.

This check exists to catch domain changes the moment they happen. Whether the change was intentional (a planned migration) or accidental (a configuration management error), you need to know immediately so you can coordinate the update across all dependent systems before the silent failures compound.

Common causes

  • Configuration management drift. Infrastructure-as-code tools (Terraform, Ansible, Helm) push a config change that modifies the jetstream { domain } value. This often happens when templates are updated without awareness that the domain is a cross-system dependency, not just a local server setting.

  • Server config file edited manually. An operator edits the server configuration and changes or removes the domain value, then reloads the config. The server accepts the change, but no one updates the downstream clients and leafnode configurations.

  • Environment promotion error. A staging configuration with domain: "staging" is accidentally deployed to production servers that should have domain: "prod". All leafnode clients referencing $JS.prod.API.> immediately lose JetStream access.

  • Rolling upgrade with config change. During a rolling server upgrade, a new config template is applied that includes a domain change. Some servers run the old domain while others run the new one, creating a split where JetStream API availability depends on which server a leafnode connects to.

  • Intentional migration without coordination. The domain is changed deliberately to reorganize the JetStream topology, but downstream clients and mirror/source configurations are not updated simultaneously. The gap between the domain change and the client updates is the danger window.

How to diagnose

Confirm the domain change

Check the current JetStream domain on the affected server:

Terminal window
nats server info

Look for the JetStream Domain field. Compare this value with what your configuration management system expects.

Check which servers are affected

If you have multiple servers, verify domain consistency across the cluster:

Terminal window
nats server list --json | jq '.[] | {name: .name, domain: .jetstream.domain}'

All servers in the same JetStream deployment should report the same domain value (or no domain, if they don’t use leafnode-scoped access).

Verify leafnode clients can still reach JetStream

From a leafnode-connected client, test JetStream API access using the expected domain:

Terminal window
nats account info

If this returns a timeout or “no responders” error from a leafnode connection, the domain prefix is likely wrong. The client is sending requests to an API prefix that no longer has a listener.

Check for broken mirrors and sources

Streams that mirror or source from a domain-prefixed origin will stop receiving data if the domain changed:

Terminal window
nats stream info <stream-name>

Look at the Mirror or Sources section. If the External API Prefix references the old domain, the mirror/source is broken. The Lag value will grow continuously.

Review server logs for the change

The NATS server logs the effective configuration at startup and reload. Search for the domain value:

Terminal window
grep -i "domain" /var/log/nats/nats-server.log

How to fix it

Immediate: verify intent and assess blast radius

Determine whether the change was intentional. Check your change management records, CI/CD pipeline history, and configuration management state. If the change was accidental, revert the domain to its previous value and reload the server configuration:

Terminal window
nats-server --signal reload

Identify all systems that reference the old domain. This includes:

  • Client applications using nats.JetStreamDomain() or equivalent
  • Stream mirrors and sources with External.ApiPrefix
  • Leafnode configurations that route JetStream API traffic

Short-term: update all references

If the domain change is intentional, update all dependent configurations to reference the new domain.

Update client applications. Every client that connects via leafnode and accesses JetStream through a domain prefix must be updated:

1
// Go client — update the domain option
2
js, err := nc.JetStream(nats.Domain("new-domain"))
3
if err != nil {
4
log.Fatal(err)
5
}
1
# Python (nats.py) — update the domain option
2
js = nc.jetstream(domain="new-domain")

Update mirror and source configurations. For each stream that mirrors or sources from the changed domain:

Terminal window
# Check current mirror config
nats stream info ORDERS-MIRROR --json | jq '.config.mirror'
# Update the mirror's external API prefix. There is no --mirror-external-api
# flag on natscli — fetch the JSON config, patch the mirror.external prefix,
# and feed it back via `nats stream edit --config <file>`:
nats stream info ORDERS-MIRROR --json | jq '.config |
.mirror.external = {api: "$JS.new-domain.API", deliver: ""}' > /tmp/orders-mirror.json
nats stream edit ORDERS-MIRROR --config /tmp/orders-mirror.json

Update leafnode remote configurations. If your leafnode configs explicitly map account subjects, verify the JetStream API prefix routing is correct in the leafnode server configuration.

Long-term: prevent accidental domain changes

Lock the domain value in configuration management. Treat the JetStream domain as a critical infrastructure constant — equivalent to a database name or Kubernetes namespace. Pin it in your IaC templates and add validation that the domain value matches the expected value.

Add a pre-deploy check. Before deploying server configuration changes, diff the effective config and flag any change to the jetstream.domain field for manual review:

Terminal window
# Compare current domain with intended
CURRENT=$(nats server info --json | jq -r '.info.domain // empty')
EXPECTED="hub"
if [ "$CURRENT" != "$EXPECTED" ]; then
echo "WARNING: JetStream domain mismatch: $CURRENT != $EXPECTED"
fi

Document the domain topology. Maintain a clear document or diagram showing which servers use which domain, which leafnodes connect to them, and which clients reference each domain. This makes the blast radius of any change immediately visible.

Frequently asked questions

What happens if I change the JetStream domain while clients are connected?

Existing connections are not dropped — the server doesn’t disconnect clients when the domain changes. However, any new JetStream API request that uses the old domain prefix will fail silently (no responders). Clients will experience timeouts on publish acknowledgments, stream info requests, and consumer operations that route through the old domain prefix. The failure is silent because the old API subject simply has no listener anymore.

Do I need a JetStream domain if I’m not using leafnodes?

No. The JetStream domain is only relevant for cross-boundary JetStream access via leafnodes. If all your clients connect directly to the cluster (or via simple routes/gateways), the domain setting has no effect on JetStream API routing. You can leave it unset.

Can two clusters have the same JetStream domain?

No — within a connected topology, each JetStream deployment must have a unique domain. If two clusters connected via leafnodes share the same domain name, API requests will be routed ambiguously and may reach the wrong cluster’s JetStream. Use distinct domain names for each cluster that needs to be independently addressable.

How does Insights detect the domain change?

Insights compares the jetstream.config.domain value reported in each server’s /varz endpoint across consecutive collection epochs. If the value changes between two epochs — whether from empty to set, from one value to another, or from set to empty — the check fires. This catches domain additions, removals, and modifications.

Should I coordinate domain changes across all servers simultaneously?

Yes. During the window where some servers have the old domain and others have the new one, leafnode clients may get inconsistent JetStream API routing depending on which server they connect to. Plan domain changes as a coordinated operation: update the configuration on all servers, then trigger a simultaneous reload or rolling restart within a short window.

Proactive monitoring for NATS jetstream domain changed with Synadia Insights

With 100+ always-on audit Checks from the NATS experts, Insights helps you find and fix problems before they become costly incidents.
No alert rules to write. No dashboards to maintain.

Start a 14-day Insights trial
Cancel