Checks/OPT_SYS_006

NATS Leaf Compression Disabled: What It Means and How to Fix It

Severity
Info
Category
Consistency
Applies to
System Improvement
Check ID
OPT_SYS_006
Detection threshold
Leafnode connection has compression disabled

A leafnode connection with compression disabled is transmitting all NATS protocol data — messages, subscriptions, and control frames — uncompressed between the leaf server and the hub. Since NATS 2.10, leafnode connections support S2 compression, which typically reduces bandwidth usage by 40-70% on text-heavy message payloads with negligible CPU overhead.

Why this matters

Leafnode connections frequently span network boundaries where bandwidth is metered and expensive: edge to cloud, on-premises to public cloud, between cloud regions, or across availability zones. Every byte transmitted on these links costs money — cloud egress charges, dedicated circuit billing, or cellular data in IoT deployments. Compression directly reduces the byte volume on these connections, translating to lower network costs for the same message throughput.

Beyond cost, compression improves effective throughput on bandwidth-constrained links. A leafnode connection over a 10 Mbps WAN link can carry roughly 1 MB/s of NATS traffic uncompressed. With 60% compression, that same link supports 2.5 MB/s of logical message throughput. For deployments where adding bandwidth is impractical — remote sites, satellite links, mobile backhaul — compression is the simplest way to increase capacity without infrastructure changes.

The performance overhead of S2 compression is minimal. S2 (Snappy-2) is specifically designed for high-speed compression with low CPU cost. In benchmarks, S2 compression adds less than 1% CPU overhead on modern hardware while achieving compression ratios of 2-4x on typical NATS workloads (JSON, Protocol Buffers, structured text). The s2_auto mode dynamically adjusts the compression level based on connection RTT, using lighter compression on low-latency local links and heavier compression on high-latency WAN links.

Common causes

  • Default configuration does not enable compression. NATS servers do not enable leafnode compression by default. If the server configuration was written before NATS 2.10 or was generated from a template that doesn’t include the compression block, leafnode connections run uncompressed.

  • Server upgraded without updating configuration. The NATS binary was upgraded to 2.10+ (which supports compression), but the server configuration file was not updated to include the compression settings. The capability exists but isn’t activated.

  • Configuration template missing compression block. Infrastructure automation (Terraform, Ansible, Helm charts) generates server configs from templates. If the template predates compression support or the compression parameter wasn’t added, all deployed servers lack it.

  • Compression intentionally disabled for debugging. An operator disabled compression during a troubleshooting session to rule it out as a source of issues and forgot to re-enable it.

  • Mixed-version deployment. Some servers in the deployment run NATS 2.10+ (compression supported) while others run older versions. Compression cannot be negotiated with pre-2.10 servers, so it remains off for those connections.

How to diagnose

Check leafnode compression status

Use the monitoring endpoint to inspect compression on leafnode connections:

Terminal window
curl -s http://localhost:8222/leafz?subs=false | jq '.leafs[] | {name: .name, account: .account, compression: .compression, rtt: .rtt}'

Any connection showing "compression": "off" or missing the compression field entirely is running without compression.

Check from the server info

Terminal window
nats server info

Review the leafnode connections section. Compression status is reported per-connection.

Audit all leafnodes across the deployment

To check every server’s leafnode compression status:

Terminal window
nats server list --json | jq -r '.[].name' | while read server; do
echo "=== $server ==="
curl -s "http://$server:8222/leafz?subs=false" 2>/dev/null | jq '.leafs[] | {name: .name, compression: .compression}' 2>/dev/null
done

Verify server version supports compression

Compression requires NATS Server 2.10.0 or later on both sides of the connection:

Terminal window
nats server list --json | jq '.[] | {name: .name, version: .version}'

If any leafnode server runs a version prior to 2.10, compression cannot be enabled for that connection.

How to fix it

Immediate: enable compression

Enable S2 compression in the leafnode configuration. The recommended mode is s2_auto for adaptive compression based on RTT. Available modes: s2_fast, s2_better, s2_best, s2_auto (default when enabled). Configure on both hub and leaf sides:

1
# Hub server configuration
2
leafnodes {
3
port: 7422
4
compression: {
5
mode: s2_auto
6
}
7
}
1
# Leaf server configuration
2
leafnodes {
3
remotes [
4
{
5
urls: ["nats-leaf://hub.example.com:7422"]
6
compression: {
7
mode: s2_auto
8
}
9
}
10
]
11
}

Apply the configuration with a reload — no restart required:

Terminal window
nats server config reload <server-id>

Or send a SIGHUP to the NATS process:

Terminal window
kill -HUP $(pidof nats-server)

Compression is negotiated during connection setup. Existing leafnode connections will be re-established with compression after the reload.

Short-term: choose the right compression level

S2 offers multiple modes. Choose based on your environment:

ModeCPU CostCompression RatioBest For
s2_autoAdaptiveAdaptiveMost deployments — adjusts to RTT
s2_fastVery low~2xLow-latency local connections
s2_betterLow~3xWAN connections with moderate bandwidth
s2_bestModerate~4xBandwidth-constrained links (satellite, cellular)

For most deployments, s2_auto is the right choice. It uses RTT measurements to select the compression level: lighter compression on fast links (where CPU savings matter more) and heavier compression on slow links (where bandwidth savings matter more).

Per-remote compression configuration lets you set different levels for different leaf connections:

1
leafnodes {
2
remotes [
3
{
4
urls: ["nats-leaf://local-hub:7422"]
5
compression: { mode: s2_fast } # Same datacenter
6
},
7
{
8
urls: ["nats-leaf://remote-hub:7422"]
9
compression: { mode: s2_better } # Cross-region
10
}
11
]
12
}

Long-term: standardize compression in all configurations

Update infrastructure templates. Add the compression block to your Helm charts, Terraform modules, Ansible roles, or whatever generates NATS server configurations. This ensures every new deployment gets compression by default:

1
// Go — programmatic config generation example
2
leafConfig := map[string]interface{}{
3
"port": 7422,
4
"compression": map[string]interface{}{
5
"mode": "s2_auto",
6
},
7
}
1
# Python — config generation example
2
leaf_config = {
3
"leafnodes": {
4
"port": 7422,
5
"compression": {
6
"mode": "s2_auto"
7
}
8
}
9
}

Monitor compression ratios. After enabling compression, track the bandwidth reduction to quantify the benefit. Compare bytes in/out on leafnode connections before and after to validate the expected savings.

Combine with application-level compression for maximum benefit. If messages are already compressed (gzip’d payloads, compressed Protobuf), S2 won’t achieve much additional compression on the message bodies. In these cases, S2 still compresses the NATS protocol overhead (headers, subject lines, reply subjects), which provides some benefit on high-message-rate connections even with pre-compressed payloads.

Frequently asked questions

Does leafnode compression require both sides to be configured?

Compression is negotiated between the hub and the leaf. Both sides must support compression (NATS 2.10+), and at least one side must have it enabled. If one side has compression enabled and the other doesn’t explicitly configure it, the negotiation will determine whether compression is used. For reliable behavior, configure compression on both the hub and leaf server.

How much bandwidth does S2 compression save on NATS leafnode connections?

It depends on message content. JSON and structured text typically compress 60-70%. Protocol Buffers and other binary formats compress 30-50%. Already-compressed payloads (images, encrypted data) see minimal benefit on the payload itself, but NATS protocol overhead is still compressed. For a typical mixed workload, expect 40-60% bandwidth reduction.

Does compression add latency to message delivery?

Negligible. S2 compression in s2_fast mode operates at multiple GB/s on modern CPUs — compressing a 1KB message takes microseconds. The latency added by compression is orders of magnitude smaller than the latency saved by transmitting fewer bytes over the network, especially on WAN connections. For same-datacenter links with sub-millisecond RTT, the difference is immeasurable.

Can I enable compression on route and gateway connections too?

NATS 2.10+ supports compression negotiation on routes and gateways as well. However, the configuration and behavior differ from leafnodes. Route compression is automatically negotiated between peers that support it. This check specifically targets leafnode connections, which are the most common candidate for compression due to their typical placement on bandwidth-constrained WAN links.

Will enabling compression cause a leafnode reconnection?

Yes. When you reload the server configuration with compression enabled, existing leafnode connections are renegotiated. This causes a brief reconnection — typically under a second. During this window, messages may be buffered. The reconnection is automatic and transparent to clients connected to the leaf server, but plan the change during a maintenance window if zero-downtime is critical.

Proactive monitoring for NATS leaf compression disabled with Synadia Insights

With 100+ always-on audit Checks from the NATS experts, Insights helps you find and fix problems before they become costly incidents.
No alert rules to write. No dashboards to maintain.

Start a 14-day Insights trial
Cancel