NATS Meta Raft Group Closed

Every JetStream-enabled server runs a Raft group named _meta_ in the system account. That group is the JetStream control plane: it holds the assignment of every stream and consumer to servers, and every create, delete, scale, and placement decision is a proposal into its log. This check fires when a server’s own Raft report shows _meta_ in state CLOSED while the server is still answering monitoring requests.

Why this matters

A closed meta group is a control-plane outage on one server that does not look like an outage. The process is up, it accepts client connections, it answers varz and healthz, and any stream whose own Raft group still has a leader keeps publishing and consuming. Nothing in a dashboard built on connection counts or message rates changes. What has stopped is the server’s participation in JetStream cluster decisions.

The immediate consequence is that the server can no longer vote in meta elections or apply meta log entries. In a three-server cluster, the meta group needs two members to hold quorum. One closed meta group takes the cluster from tolerating one failure to tolerating none: the next server to restart, lose its network, or run out of disk leaves the meta group without a majority, and at that point stream and consumer creation, deletion, and placement stop cluster-wide. The finding is critical because of that lost margin, not because the cluster is already down.

The second consequence is drift. Assets assigned to this server before the meta group closed are still assigned to it in the cluster’s view. If a stream replica on this server needs to be moved, a consumer needs to be created against a stream it leads, or a peer needs replacing, the meta leader will propose it and this server will never apply it. The longer the group stays closed, the further its local state falls behind the assignments the rest of the cluster believes are in force.

A closed Raft node does not reopen. nats-server closes it deliberately, on a fatal write error, a failed store recovery, or an explicit shutdown, and there is no retry loop that brings it back. The group stays closed for the life of the process, which is why this check treats a single observation as actionable rather than waiting for the condition to persist.

The check looks only at _meta_. Stream and consumer Raft groups close as a normal part of deleting the asset they back, so a closed asset group is expected and is not reported here.

Common causes

• A fatal write error on the meta group’s write-ahead log. nats-server filters the benign store conditions and treats everything else as fatal: it logs Critical write error, records the error on the group, and shuts the node down in the same step. A full disk, an exhausted inode table, a read-only remount, or a permissions change on the JetStream store directory all land here. The Raft WAL Write Error check (server-064) reports this cause directly and names the group.

• JetStream was disabled at runtime. An out-of-space error anywhere in the Raft layer calls the server’s out-of-space handler, which disables JetStream for the whole server. The meta node is shut down as part of that, but the server itself keeps running and keeps serving core NATS traffic.

• Meta store recovery failed at startup. The meta group’s log lives under <store_dir>/$SYS/_js_/_meta_. If those files are truncated, partially written, or unreadable after an unclean shutdown or a host-level filesystem repair, recovery fails and the node never opens.

• The server was removed from the meta group as a peer. Running nats server cluster peer-remove against a server that is still up removes it from the meta group’s peer set. The server continues to run without a working meta node until it is restarted and reassigned.

• The JetStream store directory is not where the server expects it. An unmounted volume, a container restart that lost a bind mount, or a config reload that changed store_dir leaves the meta node writing into a path that fails on the first write.

• The server is mid-shutdown. During a lame duck drain or a partial shutdown the meta node is closed before the process exits. Observing this once at the tail of a planned restart is expected; observing it on a server that has been up for hours is not.

How to diagnose

Confirm the state on the server itself

The /raftz endpoint reports every Raft node the server is running. With no account filter it returns the system account, which is where _meta_ lives:

curl -s "http://<server_host>:8222/raftz" | jq '."$SYS"."_meta_" | {state, leader, term, committed, applied, wal_error}'

A state of CLOSED is the condition this check reports. Note the value of wal_error at the same time: if it is set, the group was closed by a write failure and the fix is a storage fix, not a restart on its own.

Ask the server how it sees its own JetStream layer

healthz reports the meta layer separately from asset health, so you can isolate it:

curl -s "http://<server_host>:8222/healthz?js-meta-only=true&details=true" | jq

A closed meta node returns JetStream meta layer is not running. If instead the error reads JetStream meta layer write error: ..., the operating system’s error is in that string and tells you what failed.

Over the system account, the same request without shell access to the host:

nats server request healthz --name <server_name>

Check how the rest of the cluster sees the meta group

nats server report jetstream

The meta group table lists every server, which one is the meta leader, and each member’s lag. A server whose meta node is closed shows as offline or absent from the group. Compare the number of members still current against the quorum the group needs: for three peers that is two, for five it is three.

nats server list

Confirms the server is still in the cluster and answering. A server that shows up here but not as a healthy meta peer is exactly the split this check describes.

Find the reason in the server log

The shutdown reason is logged once, at the moment it happens, and is not repeated:

journalctl -u nats-server --since "-24h" | grep -Ei "critical write error|meta layer|jetstream disabled|recovering"

In Kubernetes:

kubectl logs <nats_pod> --since=24h | grep -Ei "critical write error|meta layer|jetstream disabled|recovering"

Critical write error points at storage. A message about disabling JetStream points at the out-of-space handler. Recovery errors at startup point at the meta store files.

Check the JetStream store directory

# Free space and inodes on the volume holding the store directory
df -h /var/lib/nats/jetstream
df -i /var/lib/nats/jetstream

# Ownership, permissions and whether the mount is read-only
ls -ld /var/lib/nats/jetstream
mount | grep /var/lib/nats

# The meta group's own log
ls -l /var/lib/nats/jetstream/\$SYS/_js_/_meta_

Substitute the store_dir from the server’s JetStream configuration. nats server info <server_name> reports the configured JetStream paths and limits if you are not sure which path is in use.

How to fix it

Immediate: fix the underlying fault, then restart the server

A restart is what reopens the meta node, but a restart on top of an unfixed storage fault closes it again within seconds. Clear the cause first.

If wal_error or the log points at storage, free space or fix permissions before restarting:

# Reclaim space on the JetStream volume, then confirm
df -h /var/lib/nats/jetstream

# Confirm the server's user can write where it is configured to
sudo -u nats touch /var/lib/nats/jetstream/.write_test && sudo -u nats rm /var/lib/nats/jetstream/.write_test

Then restart one server at a time:

sudo systemctl restart nats-server

In Kubernetes, delete the pod and let the StatefulSet recreate it:

kubectl delete pod <nats_pod>

Watch the meta group come back before touching any other server:

watch -n 2 'nats server report jetstream'

The server should reappear as a meta peer with lag falling to zero. Confirm on the server itself:

curl -s "http://<server_host>:8222/raftz" | jq '."$SYS"."_meta_".state'

A healthy node reports LEADER or FOLLOWER.

Short-term: repair a meta store that will not recover

If the server restarts and the meta node still fails to open, the meta log on this server is unusable. The meta group’s state is fully replicated, so a healthy majority elsewhere can rebuild this member from scratch. Remove the peer, stop the server, delete only the meta group’s directory, and bring it back:

# From a server that is still in the meta group
nats server cluster peer-remove <server_name>

# On the affected host, with the server stopped
sudo systemctl stop nats-server
sudo rm -rf /var/lib/nats/jetstream/\$SYS/_js_/_meta_
sudo systemctl start nats-server

Delete only the _meta_ directory. The account directories beside it hold stream data, and removing them destroys replicas the cluster is counting on. If the meta group elsewhere does not currently have quorum, do not do this at all: recover quorum first, because a rebuilt member needs a leader to catch up from.

Long-term: make the failure visible and less likely

Alert on the meta group, not on the process. A closed meta node leaves every process-level and connection-level signal healthy. Health checks that only test whether the port answers will pass throughout the incident. Probe the meta layer explicitly:

1
readinessProbe:
2
  httpGet:
3
    path: /healthz?js-meta-only=true
4
    port: 8222
5
  periodSeconds: 10

Give the JetStream store its own volume with headroom. Sharing the store directory with logs, container images, or another service means an unrelated process can fill the disk and take the meta group down with it. Alert on the volume at a threshold that leaves time to act, not at the point where writes already fail.

Watch for the fault that precedes this one. Storage saturation and WAL write errors are the upstream conditions. The JetStream Storage vs Configured Limit and Raft WAL Write Error checks fire earlier in the same chain, and acting on them avoids reaching this state.

Restart servers one at a time and verify between each. After every restart, wait until nats server report jetstream shows the server back in the meta group with zero lag before moving on. A rolling restart that does not wait can close one meta node while another has not yet rejoined.

Frequently asked questions

Do clients see errors when a server’s meta group closes?

Usually not right away. Core NATS publish and subscribe are unaffected, and any stream whose Raft group still has a leader keeps serving. What fails is JetStream management: creating or deleting a stream or consumer, changing replica counts, and moving placement. Clients that only publish and consume may notice nothing until a second failure removes the meta group’s quorum.

Is this the same thing as Meta Quorum Lost?

No, and the order matters. This check reports one server whose meta node has closed while the rest of the group may still be healthy. Meta Quorum Lost (system-007) reports that the group as a whole no longer has a majority and the control plane is down cluster-wide. A closed meta node on one server is what erodes the margin that keeps the second condition from happening.

Why does the server keep reporting if its meta group is dead?

Because the Raft node and the server process are independent. The monitoring endpoints, the client accept loop, and the route connections all live outside JetStream. nats-server closes a Raft node when it cannot safely continue writing to its log, and closing that node does not stop the server. That is what makes this check necessary: the server’s liveness signals stay green.

Will the meta group reopen on its own if the disk frees up?

No. The node is closed, not paused, and nothing reopens a closed Raft node in a running process. Freeing space is necessary but not sufficient; the server has to be restarted for a new meta node to be created and recovered.

Can I delete the whole JetStream store directory instead of only _meta_?

You can, but only if you accept losing every replica this server holds. The meta group’s state is replicated and cheap to rebuild from a healthy majority. Stream data is not: deleting an account directory removes real replicas, and the cluster will have to re-replicate every one of them from their remaining peers, which costs time and network and reduces redundancy while it runs. Remove _meta_ alone unless you are deliberately rebuilding the server.