NATS Server Not Reporting

A server is reported as not reporting when monitoring collected no VARZ response from it during a collection cycle in which at least one other server did answer VARZ, and at least one peer in its cluster still reported an open route to it. The server has not been declared down. Something between the monitoring request and a VARZ response is missing, and this finding says so without claiming to know which.

Why this matters

An absent VARZ response is ambiguous on its own. A server that has genuinely stopped answers nothing, and so does a server that was never asked. This check resolves part of that ambiguity by requiring two independent pieces of evidence before it fires: another server answered VARZ in the same cycle, so the collection itself was not skipped; and a cluster peer reported a live route to this server, so the cluster still considers it a member.

What remains after those two conditions is a server the cluster believes is connected but that monitoring cannot see. That gap matters because most of what you would use to answer “is this server healthy” comes from VARZ: uptime, connection counts, subscription counts, memory, route and gateway state, slow consumer counters, configuration as loaded. Without it, every check that depends on those fields goes quiet for that server, and quiet reads as healthy.

Not everything goes quiet. STATSZ is delivered through a separate internal queue inside nats-server, so CPU, traffic, storage, and the other server measurements it carries can keep arriving while VARZ does not. Checks built on those values can still fire for a server reported here. That asymmetry is the most useful clue this finding gives you, and it is the reason the check does not treat a STATSZ response as evidence that the server is reporting.

Coverage is limited by design. Route evidence is intra-cluster, so a solo server, a single-member cluster, or a node reachable only over gateways carries no route evidence and is never reported here. The finding also fires only in cycles where route data was collected, because a missing route observation cannot be substituted from history.

The finding does not identify the cause. It tells you where to look.

Common causes

• The monitoring server filter excludes it. A deployment can restrict which discovered servers are scraped by server name, cluster name, or tag. A server that was being scraped and then matched a deny entry, or stopped matching an allow entry, receives no monitoring requests at all while its peers keep routing to it normally. Nothing in the catalog records which servers a filter excluded, so this looks identical to silence.

• System account permissions changed. Monitoring requests travel on $SYS.REQ.SERVER.* subjects in the system account. A credential rotation, a tightened permission set, or an account JWT update that narrows the allowed subjects stops the requests or their replies without stopping anything else the server does.

• The shared system receive queue is blocked. nats-server dispatches VARZ, ROUTEZ, HEALTHZ, JSZ, and CONNZ through one internal queue and STATSZ through its own. A monitoring handler that blocks, most often on a large JetStream operation, delays everything behind it on the shared queue. The server keeps routing traffic and keeps publishing STATSZ while VARZ requests sit unanswered.

• The request timed out rather than failed. A server under heavy load can answer VARZ correctly but later than the monitoring request was willing to wait. Large connz and subsz responses on a server with very high connection or subscription counts make this more likely.

• A restart that monitoring has not reconciled yet. A restarted server comes back with a new identity, and a peer can hold a route to the previous one for a cycle or two. This check suppresses the old identity once the successor is observed in the same cluster, but a cycle captured mid-restart can still produce a finding.

• An asymmetric network path. The route is established and the peer reports it, but replies from this server are not reaching the monitoring collector. A one-way firewall rule, an MTU problem, or a partial partition produces this.

How to diagnose

Establish whether the server is serving traffic

Start by separating “monitoring cannot see it” from “it is broken”. The peer count in the finding tells you how many distinct peers reported a route to it at that moment. Confirm from the cluster side:

nats server list

If the server appears here, it is answering on the STATSZ path, which is the separate queue. That already rules out a fully dead process and points at a monitoring or scrape problem. Check its routes from a peer:

curl -s 'http://<peer_host>:8222/routez' | jq '.routes[] | select(.remote_name == "<server_name>") | {remote_name, rtt, last_activity, pending_size}'

A recent last_activity and a normal rtt mean the cluster link is healthy.

Check the monitoring server filter

If the deployment restricts which servers are scraped, confirm this server still matches. The filter matches exactly on server name, cluster name, or tag, and deny entries take precedence over allow entries:

1
scraper:
2
  filter:
3
    allow:
4
      servers: []
5
      clusters: []
6
      tags: []
7
    deny:
8
      servers: []
9
      clusters: []
10
      tags: []

An allow list that was written for an earlier topology is the common case: new servers joined, the list was not updated, and they are silently excluded. Compare the list against what the cluster actually reports:

nats server list --json | jq -r '.[].server.name' | sort

Check system account permissions and request errors

Monitoring runs as a client in the system account. Verify the credentials it uses can still make the request, using those same credentials:

nats --creds=<monitoring_creds> server request variables --name=<server_name>

A permissions violation shows up in the server log on the monitoring connection’s side:

grep -E 'Permissions Violation|Authorization Error' /var/log/nats/nats-server.log

Also check the collector’s own logs for request errors and timeouts against this server name, which distinguish a refused request from one that was never sent.

Request VARZ directly and compare against other endpoints

Ask the server for VARZ over the same connection monitoring uses, then ask for the other endpoints and see which answer:

# VARZ, over the shared internal queue
nats --creds=<monitoring_creds> server request variables --name=<server_name>

# ROUTEZ and HEALTHZ, also on the shared queue
nats --creds=<monitoring_creds> server request routes --name=<server_name>
nats --creds=<monitoring_creds> server request jetstream-health --details --name=<server_name>

# STATSZ, on its own queue
nats --creds=<monitoring_creds> req '$SYS.REQ.SERVER.PING.STATSZ' '' --replies=0 --timeout=5s

If the server’s monitoring port is reachable, the same comparison over HTTP removes the system account from the picture entirely:

curl -s --max-time 5 http://<server_host>:8222/varz | jq '{name, start, connections, routes, cpu, mem}'
curl -s --max-time 5 http://<server_host>:8222/healthz

Two distinct outcomes matter here. VARZ answering over HTTP but not over NATS points at the system account or the filter. VARZ hanging on both, while STATSZ keeps arriving, points at the shared queue inside the server.

Take a goroutine profile if monitoring hangs while STATSZ responds

PROFILEZ is handled in its own goroutine rather than on the shared queue, so it still answers when VARZ does not. That makes it the right tool for exactly this state:

mkdir -p ./profiles
nats --creds=<monitoring_creds> server request profile goroutine ./profiles --level=2 --name=<server_name>

Read the profile and look for the system message handler goroutines. What you are looking for is a monitoring handler parked on a lock or on a stream operation, with the queue behind it not draining:

go tool pprof -traces ./profiles/<profile_file>

Blocked stream or consumer operations in those stacks explain why a shared handler is slow, and they are also what you report if the block does not clear.

How to fix it

Immediate: decide whether the server is degraded or only unobserved

Answer one question first. If STATSZ values are still arriving and CPU, traffic, and storage look normal, and clients connected to the server are working, the server is serving and the problem is in the monitoring path. Do not restart it. Fix the collection.

If monitoring requests hang, STATSZ values are flat or missing, or clients on that server are seeing errors, treat it as degraded. Move JetStream leadership off it before it becomes a quorum problem:

nats server report jetstream --host=<server_host>
nats stream cluster step-down <stream_name>

Capture a goroutine profile before restarting, because the restart destroys the only evidence of what was blocked.

Short-term: correct the collection path or restart the server

If the filter excluded it, add the server, cluster, or tag back to the allow list or remove it from the deny list, then reload the collector’s configuration. The server should appear in the next cycle.

If credentials or permissions are the cause, restore the monitoring user’s access to the $SYS.REQ.SERVER.> subjects in the system account and reconnect the collector. Verify with a single request before assuming it is fixed:

nats --creds=<monitoring_creds> server request variables --name=<server_name>

If the request is timing out, raise the collection timeout rather than lowering what is collected. A server with very high connection or subscription counts legitimately takes longer to build a VARZ response.

If the shared queue is blocked, restart the server once you have the profile:

sudo systemctl restart nats-server

kubectl delete pod <nats_pod_name>

Confirm the server reports again on the following cycle and that its route count matches its peers.

Long-term: keep the gap from reopening

Keep the server filter in sync with the topology. A filter written by hand against a fixed server list goes stale the first time the cluster scales. Prefer tag-based or cluster-based selection, which survives servers joining and leaving.

Alert on this finding rather than only on server health. A server nobody can see reads as healthy in every check that depends on VARZ. That is the failure mode this check exists to make visible, so it needs to reach someone.

Watch the endpoints that share the queue. ROUTEZ, HEALTHZ, JSZ, and CONNZ go silent with VARZ. Tracking whether they answer, alongside STATSZ which does not share the queue, turns the next occurrence into a one-step diagnosis.

Pace rolling restarts. Restarting a whole cluster faster than one collection cycle produces findings for identities that no longer exist, alongside the ones that matter. Waiting for each server to report before restarting the next keeps the signal clean.

Give the system account monitoring user stable, reviewed permissions. Credential rotation is a routine operation that silently breaks observability when the replacement permissions are narrower than the originals. Verify a VARZ request after every rotation.

Frequently asked questions

Does this mean the server is down?

No. A server that is genuinely down is reported by the checks that track membership and restarts. This finding is narrower: monitoring collected nothing from the server in a cycle where it collected from others, and a peer still reported a route to it. The server is very often running and serving clients normally while this fires.

Why is my single-node server never reported?

The check requires route evidence, and routes exist only between servers inside a cluster. A solo server, a single-member cluster, and a node reachable only over gateways have no peer that can report a route, so there is nothing to distinguish “silent” from “not asked”. Those deployments need a different signal, such as monitoring the server’s own healthz endpoint directly.

The server shows up in nats server list, so why is it reported?

nats server list is built from the STATSZ path, which nats-server dispatches through its own internal queue. VARZ, ROUTEZ, HEALTHZ, JSZ, and CONNZ share a different queue. A blocked handler on the shared queue silences VARZ while STATSZ keeps answering, so a server can be listed and still be invisible to everything that depends on VARZ. That combination is a strong signal, not a contradiction.

What does the peer count in the finding tell me?

It is the number of distinct peers that reported an open route to this server in the evaluated cycle. A count matching the rest of the cluster means the server is fully connected and the problem is in the monitoring path. A count well below its peers means the server is also losing routes, which moves the investigation toward the network or the server’s own health.

A rolling restart produced these findings. Is that expected?

Briefly, yes. A restarted server comes back with a new identity while peers can still hold a route to the previous one for a cycle or two. The check suppresses the old identity as soon as the successor is observed in the same cluster, so findings from a restart clear on their own. Findings that persist across several cycles are not restart artifacts and should be investigated.