Checks/SERVER_006

NATS JetStream Domain Whitespace: What It Means and How to Fix It

Severity
Warning
Category
Consistency
Applies to
Server
Check ID
SERVER_006
Detection threshold
JetStream domain name contains whitespace characters

JetStream domain whitespace means a NATS server’s JetStream domain name contains leading or trailing whitespace characters — spaces or tabs that are invisible in most configuration views but cause the JetStream API to be unreachable through the expected subject prefix.

Why this matters

JetStream domains control how clients reach the JetStream API. When a domain is configured, the API subject prefix becomes $JS.<domain>.API.>. A domain value of "hub" produces $JS.hub.API.>. A domain value of "hub " (with a trailing space) produces $JS.hub .API.> — a completely different subject that no client will ever publish to intentionally.

The result is silent failure. Clients attempting to access JetStream through a leafnode or cross-domain configuration send requests to the correct, expected subject — and get no response. There’s no error message, no log warning, no connection failure. The requests simply time out. Operators spend hours debugging JetStream connectivity issues, checking network routes, leafnode configurations, and account permissions, when the root cause is a single invisible character in a configuration value.

In multi-cluster deployments using leafnodes for cross-domain JetStream access, this problem is especially damaging. The hub domain name is part of the routing path for every JetStream API request from leaf clusters. If the hub’s domain has whitespace, every leaf cluster’s JetStream access breaks simultaneously. And because the domain looks correct when you cat the config file or check the logs — whitespace at the end of a string is invisible to the human eye — the problem resists standard debugging.

Common causes

  • Copy-paste from documentation or chat. Copying a configuration snippet from a web page, Slack message, or documentation often picks up trailing whitespace. Rich text editors and web browsers are particularly prone to adding invisible characters.

  • YAML or JSON configuration without quoting. Some configuration formats treat unquoted values differently. Trailing whitespace after an unquoted YAML value may be preserved depending on the parser. Similarly, hand-edited JSON with trailing spaces inside string values will silently include them.

  • Template rendering adding whitespace. Configuration management tools (Ansible, Terraform, Helm) that render server config from templates can introduce whitespace if template variables aren’t trimmed. A Jinja2 template with domain: {{ jetstream_domain }} may produce domain: hub if the variable has trailing whitespace.

  • Editor artifacts. Some text editors add trailing whitespace when saving files, particularly after copy-paste operations or when autoformatting is enabled. The whitespace is invisible in the editor view.

  • Inconsistent domain names across servers. In a cluster, if one server’s config was edited manually while others were generated by automation, the manually edited server may have whitespace that the others don’t — causing that single server to behave differently from its peers.

How to diagnose

Check the JetStream domain for the server

Terminal window
nats server info

Look for the JetStream Domain field in the output. Unfortunately, trailing whitespace won’t be visible in the standard output format.

Reveal hidden whitespace

Use the monitoring endpoint and pipe through a tool that makes whitespace visible:

Terminal window
curl -s http://localhost:8222/varz | jq -r '.jetstream.config.domain' | cat -A

The cat -A flag shows trailing whitespace as visible characters. A clean domain shows hub$ (just the dollar sign for end-of-line). A domain with trailing whitespace shows hub $ or hub\t$.

Alternatively, check the exact bytes:

Terminal window
curl -s http://localhost:8222/varz | jq -r '.jetstream.config.domain' | xxd

Compare domains across cluster members

Terminal window
nats server list

If servers in the same cluster report different domain values, one or more may have whitespace. Since the standard output may not show the difference, query each server’s /varz endpoint directly and compare the raw domain strings.

Test JetStream API reachability

Terminal window
# This should return account info — if it times out, the domain may be wrong
nats account info

If nats account info works when connecting directly to the server but fails through a leafnode, the domain configuration on the hub side is the prime suspect.

How to fix it

Immediate: fix the configuration

Edit the server configuration and remove whitespace from the JetStream domain value. Ensure the value is quoted to prevent future whitespace issues:

1
jetstream {
2
domain: "hub"
3
store_dir: /nats/storage
4
}

Note the double quotes around hub. Quoting the domain value prevents most whitespace-introduction vectors.

Reload the server configuration without restarting:

Terminal window
nats-server --signal reload

Verify the fix:

Terminal window
curl -s http://localhost:8222/varz | jq -r '.jetstream.config.domain' | cat -A

Confirm the output shows the domain name followed immediately by $ with no space characters.

Short-term: validate all servers

Check every server in the deployment for domain whitespace. In a multi-server deployment, fix one server and miss another, and you’ll have inconsistent routing:

Terminal window
# Check all servers
for server in server1:8222 server2:8222 server3:8222; do
echo -n "$server: "
curl -s "http://$server/varz" | jq -r '.jetstream.config.domain' | cat -A
done

Restart affected clients. Clients that cached the JetStream API prefix or received errors during the misconfiguration period may need to reconnect. Most NATS client libraries will reconnect automatically, but client-side caches of JetStream API state may be stale.

Long-term: prevent recurrence

Add configuration validation to your deployment pipeline. Before deploying server configuration, validate that string fields don’t contain leading or trailing whitespace:

1
# Python — config validation in CI/CD pipeline
2
import re
3
4
def validate_nats_config(config: dict) -> list[str]:
5
errors = []
6
domain = config.get("jetstream", {}).get("domain", "")
7
if domain != domain.strip():
8
errors.append(f"JetStream domain has whitespace: '{domain}'")
9
cluster_name = config.get("cluster", {}).get("name", "")
10
if cluster_name != cluster_name.strip():
11
errors.append(f"Cluster name has whitespace: '{cluster_name}'")
12
return errors
1
// Go — config validation
2
package main
3
4
import (
5
"fmt"
6
"strings"
7
)
8
9
func validateDomain(domain string) error {
10
if strings.TrimSpace(domain) != domain {
11
return fmt.Errorf("JetStream domain contains whitespace: %q", domain)
12
}
13
return nil
14
}

Use configuration management consistently. Generate all server configurations from a single source of truth (Terraform, Ansible, Helm) rather than hand-editing individual files. This eliminates the manual editing errors that introduce whitespace.

Enable editor settings to strip trailing whitespace. Most editors and IDEs support automatic removal of trailing whitespace on save. Enable this for any file type that includes NATS server configuration.

Frequently asked questions

How can I tell if my JetStream domain has whitespace if it looks correct?

Pipe the domain value through cat -A or xxd to reveal invisible characters. The standard nats server info output and most log viewers won’t show trailing whitespace. You need a tool that explicitly renders non-printable or whitespace characters. curl -s http://localhost:8222/varz | jq -r '.jetstream.config.domain' | cat -A will show a $ immediately after the last visible character if the domain is clean, or show spaces before the $ if whitespace is present.

Does whitespace in the JetStream domain affect servers in the same cluster?

If all servers in a cluster have the same whitespace in their domain, JetStream will function correctly within that cluster — the whitespace is consistent, so subjects match. The problem surfaces when external clients, leafnodes, or other clusters reference the domain by its intended name (without whitespace). Internal consistency with external inconsistency is the worst case: everything works locally, everything fails cross-domain.

Is this the same issue as CLUSTER_004 Cluster Name Whitespace?

Same root cause — invisible whitespace in a configuration value — but different impact. Cluster name whitespace (CLUSTER_004) affects route connection identification and cluster topology. JetStream domain whitespace (SERVER_006) affects JetStream API subject routing. Both are configuration hygiene issues that cause subtle, hard-to-debug failures. If you find one, check for the other.

Can I change the JetStream domain without restarting the server?

Yes. JetStream domain is a configuration value that can be updated with a config reload (nats-server --signal reload). You do not need a full server restart. However, be aware that changing the domain on a running server changes the JetStream API subject prefix, which will disrupt any clients currently using the old prefix. See CHANGE_002 (JetStream Domain Changed) for monitoring domain changes.

Proactive monitoring for NATS jetstream domain whitespace with Synadia Insights

With 100+ always-on audit Checks from the NATS experts, Insights helps you find and fix problems before they become costly incidents.
No alert rules to write. No dashboards to maintain.

Start a 14-day Insights trial
Cancel