NATS JetStream Asset Raft Fault

A NATS server reports a JetStream asset Raft fault when a stream or consumer it is assigned to hosts has a Raft group in a state the server cannot serve from. The asset’s configuration is fine and the group may be perfectly healthy on its other replicas. The problem is local: this server’s copy of the group is skewed, unsupervised, or not current, and it will not answer for that asset until the fault is cleared.

Why this matters

This is a server-side defect, not a capacity or configuration problem. The server is running, it is in the cluster, and it is still advertising itself as a replica for the asset. What it cannot do is participate correctly in that asset’s Raft group. Depending on the exact fault, the replica silently stops making progress, stops applying committed entries, or holds a node object that no longer belongs to the running stream.

For a replicated asset the immediate effect is a loss of redundancy rather than a loss of service. An R3 stream with one faulted replica still has a majority and keeps serving, but it is now one failure away from losing quorum. If the faulted replica is also the leader, clients see elevated latency or a stalled stream until the group elects someone else.

The faults in this family do not clear on their own. A node skew means the server holds two different Raft node objects for the same group, so one of them will never receive the entries the other commits. A stopped monitor goroutine means nothing is driving that replica’s Raft loop at all. Neither condition has a retry path inside nats-server, and neither is visible from nats stream info on a healthy peer, because the healthy peers see only a replica that has fallen behind.

This check is also the catch-all for its family. Any health error on a stream, consumer, or the JetStream subsystem that the other health checks do not recognize is reported here rather than dropped. A condition introduced by a newer server version still reaches an operator, under this code, with the server’s own error text attached.

nats-server skips these checks entirely for R1 assets, so a finding here always concerns a replicated stream or consumer.

Common causes

• Cluster node skew. The server holds one Raft node for the asset’s group assignment and a different one on the running stream or consumer object. nats-server detects the mismatch, logs Detected stream cluster node skew or Detected consumer cluster node skew, and reports cluster node skew detected. This is a server bug, usually triggered by a rapid delete and recreate of the same asset, or by an assignment update that raced a restart.

• Stopped monitor goroutine. The goroutine that runs the replica’s Raft monitor loop is gone, reported as monitor goroutine not running. Without it the replica neither votes, nor appends, nor applies. A panic recovered elsewhere in the process, or a shutdown path that stopped the monitor without tearing down the assignment, leaves the group in this state.

• Group node reporting itself unhealthy. The Raft node’s own health test fails, reported as group node unhealthy. That test fails when the node is closed, has committed nothing yet, has heard nothing from the current leader within two heartbeat intervals, is still catching up, or is paused with commits it has not applied.

• Partition from the rest of the group. A route outage or an asymmetric network path stops leader heartbeats reaching this server. The replica is functioning but cannot stay current, so its health test fails. The distinguishing sign is that the group is otherwise healthy and the fault clears when the route recovers.

• A cause the other health checks do not name. Store write errors, missing assets, assignment errors, and meta layer problems each have their own check. Anything left over is reported here with the server’s raw error text, including strings introduced by a newer nats-server release.

How to diagnose

Read the health error the server is actually reporting

The finding carries the cause string, but confirm it against the server directly. The details form returns one entry per faulted asset instead of a single summary line:

nats server request jetstream-health --details --name=<server_name>

Or over HTTP against the server’s monitoring port:

curl -s 'http://<server_host>:8222/healthz?details=true' | jq '.errors[]'

Each entry carries type (STREAM, CONSUMER, or JETSTREAM), the account, stream, and consumer it applies to, and the error text. The text after is not current: is the cause this check classifies.

Inspect the asset’s Raft group on the faulted server

nats server request raft --account=<account> --group=<group_name> --name=<server_name>

Look at state, leader, ever_had_leader, committed, applied, catching_up, paused, and observer. A replica whose committed is far ahead of applied, or whose leader is empty while other peers report one, matches the group node unhealthy cause. The same data is available over HTTP:

curl -s 'http://<server_host>:8222/raftz?acc=<account>&group=<group_name>' | jq

Confirm the rest of the group is healthy

The fault is local, so check whether the asset is still being served elsewhere before you touch anything:

nats stream info <stream_name>
nats consumer info <stream_name> <consumer_name>

The cluster section names the leader and lists each replica with its current flag and lag. A group with a leader and a majority of current replicas is still serving, and you have time to work carefully.

nats server report jetstream

This shows which servers hold assets for the account and how much they are carrying, which is how you confirm the faulted server is a member of the group at all.

Check whether this server is isolated from its peers

nats server list

Compare the route count on the faulted server against its cluster peers. Then look at the routes themselves:

curl -s 'http://<server_host>:8222/routez' | jq '.routes[] | {remote_name, rtt, last_activity, pending_size}'

A short route list, a high rtt, or a stale last_activity points at a partition rather than a node-level defect. In that case the fault clears when connectivity returns and the replica catches up.

Search the server log for the skew warning

Node skew is the one cause in this family that nats-server announces in its own log at the moment it detects it:

grep -E 'cluster node skew|Healthcheck failed' /var/log/nats/nats-server.log

The surrounding lines usually show what preceded it: an asset delete, a recreate, a peer removal, or a restart. Capture those lines before restarting the server, because a restart destroys the evidence.

How to fix it

Immediate: confirm the asset is still being served

The faulted replica is one member of a group. Verify the group still has a leader and a majority elsewhere:

nats stream info <stream_name> --json | jq '.cluster | {leader, replicas: [.replicas[] | {name, current, offline, lag}]}'

If the faulted server is the leader, move leadership to a healthy replica before doing anything else:

nats stream cluster step-down <stream_name>
nats consumer cluster step-down <stream_name> <consumer_name>

If the group has already lost its majority, quorum recovery comes first and this fault second.

Short-term: restart the faulted server

A node skew and a stopped monitor goroutine do not recover without a restart, because nothing in the running process will rebuild the group’s node object. Capture the evidence first, then restart:

mkdir -p ./profiles
nats server request profile goroutine ./profiles --level=2 --name=<server_name>

sudo systemctl restart nats-server

In Kubernetes, delete the pod and let the StatefulSet replace it:

kubectl delete pod <nats_pod_name>

After the server comes back, confirm the health error is gone and the replica has caught up:

nats server request jetstream-health --details --name=<server_name>
nats stream info <stream_name> --json | jq '.cluster.replicas'

Recovery is not instant. The replica replays its write-ahead log, rejoins the group, and catches up from the leader. Watch lag fall to zero before you consider the asset fully redundant again.

Long-term: reduce the conditions that produce it

Keep server versions aligned across the cluster. Mixed versions inside one Raft group widen the window for assignment races during a rolling upgrade. Finish an upgrade across the whole cluster rather than leaving it half-applied.

Slow down asset churn. Repeated delete and recreate of the same stream or consumer name is the pattern that most often produces node skew. Where a workflow needs a clean slate, use a new name, or purge the stream instead of deleting it.

Pace rolling restarts. Wait for each server’s replicas to report current before moving to the next. Restarting into a group that is still catching up leaves more assignments in flight.

Report recurring skew upstream. A single skew is worth a restart. A skew that returns is a nats-server defect, and the server log lines plus the goroutine profile you captured are what makes it actionable. File it against the NATS server project with the version and the sequence of operations that preceded it.

Alert on the health endpoint, not only on stream state. A faulted replica in a still-serving group is invisible to nats stream info until the group loses redundancy. Polling healthz with details=true catches it at the point where you still have a majority.

Frequently asked questions

What does “cluster node skew detected” actually mean?

The server holds two different Raft node objects for the same group: one attached to the cluster assignment for the asset, and one attached to the running stream or consumer. Only one of them is being driven, so the other never sees the entries the group commits. nats-server logs a warning the moment it notices, and the replica is unusable until the process restarts and rebuilds the group from its assignment.

Is my stream down?

Not necessarily, and usually not. This fault affects one replica on one server. An R3 stream with a leader and two current replicas keeps serving normally. What you have lost is fault tolerance: the group is now one failure away from losing quorum. If the faulted server was the leader, the group elects a new one, and clients see a brief stall rather than an outage.

Why did this fire with an error message I have never seen?

This check is the catch-all for its family. Store write errors, missing assets, rejected configurations, and meta layer problems are each reported by a check that names them. Anything else, including a string a newer nats-server release introduced, is reported here with the server’s own text rather than being dropped. An unfamiliar cause is a signal to read the server’s release notes, not a sign the check misfired.

Is restarting the server safe?

For a replicated asset with a healthy majority elsewhere, yes. Step leadership down first if the faulted server leads any of the affected groups, so the restart does not force an election under load. If the group has already lost quorum, restarting removes another peer from an already broken group and makes recovery harder. Restore quorum first.

Can this fire for an R1 stream or consumer?

No. nats-server returns early from these checks when the replica count is one, so the Raft conditions are never evaluated for an unreplicated asset. A finding here always concerns a stream or consumer with more than one replica.