NATS Raft WAL Write Error

Every JetStream Raft group keeps a write-ahead log on disk. When a write to that log fails in a way nats-server cannot tolerate, the server records the error on the group and shuts the group’s Raft node down in the same step. This check reports groups carrying that error, and it names the group and the account, which the server’s own health endpoint does not.

Why this matters

The Raft log is where a group’s decisions are made durable before they are applied. If a write to it fails, the node cannot honor the guarantee it makes to its peers, so nats-server does not retry and does not degrade: it stops the node. By the time the error is visible, the group is already down on this server.

What that costs depends on which group failed. If the group is _meta_, this server has left the JetStream control plane and can no longer vote in meta elections or apply stream and consumer assignments. If the group backs a stream or a consumer, that asset is unavailable on this server. A replicated asset carries on from its other replicas, one replica short and with no fault tolerance left in a three-way group. An R1 asset has nowhere to fail over to and is down.

The error is rarely confined to one group. The most common causes, a full volume, an exhausted inode table, a revoked permission, a read-only remount, affect the whole JetStream store directory, so other groups on the same server are writing into the same failing path and will hit it on their next write. An out-of-space error goes further: nats-server responds by disabling JetStream for the entire server, which takes down every group at once. Treat one finding as the first report of a server-level storage fault, not as an isolated asset problem.

nats-server does not set this error casually. It filters out the store conditions that are part of normal operation, a closed store, an end-of-file read, a missing message, an empty pending set, a partial cache, and treats a file that does not exist as a warning rather than a fault. What remains is a real write failure from the operating system, logged as Critical write error, and the operating system’s own message is carried through to the finding.

Nothing recovers on its own. A shut-down Raft node stays shut down for the life of the process, so the server has to be restarted once the storage fault is fixed.

Common causes

• The JetStream volume is full. The most frequent cause, and the one with the widest effect: the write returns no space left on device, and nats-server disables JetStream server-wide rather than leaving assets in a half-writable state. Streams with no limits, a retention policy that never removes anything, or a Raft log that has grown without snapshotting all lead here.

• Inodes are exhausted. A volume with free bytes can still fail every create. JetStream writes many small block files per stream, so a store directory holding a large number of streams and consumers can run a filesystem out of inodes long before it runs out of space. df -h looks fine; df -i is the one that shows it.

• Permissions on the store directory changed. A package upgrade that resets ownership, a container rebuilt with a different UID, or a chown applied to the wrong path leaves the server unable to write into a directory it could write into a minute earlier. This triggers the server’s write-permission handler as well as shutting down the node.

• The filesystem remounted read-only. Linux remounts a filesystem read-only when the block layer reports errors. Every subsequent write fails instantly, so multiple groups will report an error within the same epoch.

• The underlying storage is failing or disconnected. A failing disk, an unmounted volume, a detached network block device, or an NFS or similar network filesystem that stopped responding. Network filesystems are not a supported backing store for JetStream; they add write failure modes the server treats as fatal.

• The container lost its volume mount. A restart that recreated the container without its persistent volume leaves the store directory on the container’s own filesystem, which is small and often read-only by policy. The first write that exceeds it fails.

How to diagnose

Read the error and the group from raftz

/raftz is where the error is recorded. With no account filter it returns the system account, which is where _meta_ lives:

curl -s "http://<server_host>:8222/raftz" | jq 'to_entries[] | {account: .key, groups: (.value | to_entries[] | select(.value.wal_error != null) | {group: .key, state: .value.state, wal_error: .value.wal_error})}'

For a group in a user account, pass both filters:

curl -s "http://<server_host>:8222/raftz?acc=<ACCOUNT>&group=<GROUP>" | jq

The wal_error string is the operating system’s error, and it is the single most useful piece of information in the finding. no space left on device is a full volume, or an exhausted inode table if the volume still reports free bytes. permission denied is ownership or SELinux. read-only file system is a remount. input/output error is hardware.

Ask the server’s own health check

healthz sees the same fault but does not say which group it belongs to:

curl -s "http://<server_host>:8222/healthz?details=true" | jq '.errors'

An asset group reports as node write error: <error>. The meta group reports as JetStream meta layer write error: <error>. If JetStream has already been disabled by the out-of-space handler, healthz reports that instead, which is a stronger signal than any single group.

Over the system account, without shell access to the host:

nats server request healthz --name <server_name>

Map the group name back to an asset

Raft group names are opaque: _meta_ is the control plane, S-R3F-<hash> is a three-replica file-backed stream, C-R3F-<hash> is a consumer. jsz maps them to names:

curl -s "http://<server_host>:8222/jsz?accounts=true&streams=true&consumers=true&raft=true" \
  | jq '.account_details[] | {account: .name, streams: [.stream_detail[] | {stream: .name, raft: .stream_raft_group, consumers: [.consumer_raft_groups[]? | {consumer: .name, raft: .raft_group}]}]}'

An empty account in the finding means the group is _meta_, and no lookup is needed.

Check the storage the write failed on

# Free space and free inodes on the JetStream volume
df -h /var/lib/nats/jetstream
df -i /var/lib/nats/jetstream

# Ownership and whether the mount went read-only
ls -ld /var/lib/nats/jetstream
mount | grep /var/lib/nats

# Kernel-level I/O and remount events
dmesg -T | grep -Ei "i/o error|read-only|remount|ext4|xfs" | tail -20

Substitute the store_dir from the server’s JetStream configuration. nats server info <server_name> reports the configured JetStream paths and limits.

Establish the blast radius

Before repairing anything, find out what is still serving:

# Which groups on this server have a leader, and where the leaders are
nats server report jetstream

# Whether the affected stream still has a leader elsewhere
nats stream info <stream_name>

A replicated stream whose leader is on another server is still available and can wait for a planned restart. A stream that has lost its leader, or an R1 stream on this server, is down now and sets the urgency.

How to fix it

Immediate: clear the storage fault

Nothing else works until writes succeed again. Work from the error string.

Out of space. Reclaim on the JetStream volume itself, not on /:

# Find what is consuming the volume
du -sh /var/lib/nats/jetstream/* | sort -h | tail

# Confirm the reclaim landed
df -h /var/lib/nats/jetstream

If the space is legitimately stream data, reduce it at the source rather than deleting files under the store directory. Tighten a stream’s limits so the server removes messages itself:

nats stream edit <stream_name> --max-age=168h --max-bytes=10GB

Permission denied. Restore ownership on the whole store directory to the user the server runs as:

sudo chown -R nats:nats /var/lib/nats/jetstream
sudo -u nats touch /var/lib/nats/jetstream/.write_test && sudo -u nats rm /var/lib/nats/jetstream/.write_test

Read-only file system. Do not remount read-write and move on. Check dmesg for the errors that caused it, and run the filesystem check the errors call for before putting the volume back into service.

Never delete files under the store directory to free space. Those are stream data and Raft logs. Removing them by hand corrupts assets the cluster still counts on.

Short-term: restart the server and confirm the groups come back

A shut-down Raft node is not recreated in a running process. Once writes succeed again, restart:

sudo systemctl restart nats-server

In Kubernetes:

kubectl delete pod <nats_pod>

Restart one server at a time. Then confirm no group still carries an error:

curl -s "http://<server_host>:8222/raftz" | jq '."$SYS" | to_entries[] | {group: .key, state: .value.state, wal_error: .value.wal_error}'

and that the server is back in the meta group with its assets current:

watch -n 2 'nats server report jetstream'

If a replicated stream does not recover on this server after the restart, its local replica may be unrecoverable. Remove the peer so the group can replace it from a healthy majority:

nats stream cluster peer-remove <stream_name> <server_name>
nats stream edit <stream_name> --replicas 3

Do this only after confirming the stream has a leader elsewhere. Removing a peer from a group that has already lost quorum makes the situation worse, not better.

Long-term: keep writes from failing

Give JetStream its own volume, with headroom. Sharing the store directory with logs, container images, or another service lets an unrelated process fill the disk and take JetStream down. Size the volume so normal operation sits well below capacity, and alert with enough margin to act.

Set max_file below the volume size. A JetStream storage limit that exceeds the disk means the server will happily accept data until the filesystem, rather than the server, refuses the write. Configure the limit so the server enforces it first and returns an error to the publisher instead of shutting down a Raft node:

1
jetstream {
2
  store_dir: /var/lib/nats/jetstream
3
  max_file: 100GB
4
}

Put limits on every stream. A stream with no max_age, max_bytes, or max_msgs grows until something stops it. The Streams Without Limits check reports these before they become a storage incident.

Monitor inodes as well as bytes. Most alerting covers df -h and nothing else. A store directory with many small streams and consumers can exhaust inodes on a volume that still shows free space, and the failure looks identical from inside the server.

Watch the earlier signals. JetStream Storage vs Configured Limit (server-019) and Raft WAL Size Excessive (server-051) fire while there is still room to act. This check fires after the write has already failed.

Frequently asked questions

Does the group recover when the disk is no longer full?

No. The node was shut down, not paused, and nothing in a running server reopens a closed Raft node. Freeing space is necessary but not sufficient: the server has to be restarted before a new node is created for the group.

Why does this check exist when healthz already reports the write error?

Because healthz does not say which group failed. It reports node write error for an asset and JetStream meta layer write error for the meta group, and neither names the group or the account. Reading the error where it is recorded, on the Raft group itself, means a finding tells you exactly which group stopped and which account it belongs to, so you can decide whether it is the control plane or one stream before you start digging.

Is this the same as a stream store write error?

No. Two different stores can fail. This check reports a failure to write a Raft group’s log, which is what makes the group’s decisions durable. The JetStream Store Write Error check (server-055) reports a failure in a stream’s own message store. They usually share a root cause, because both live under the same store directory, but they fail independently and a fix for one does not imply the other is healthy.

Can JetStream stay up if only one group’s log fails?

Sometimes. A permission error or an I/O error scoped to one path can take down a single group and leave the rest writing normally. An out-of-space error cannot: the server responds to it by disabling JetStream server-wide. Check healthz before assuming the damage is limited to the group in the finding.

What happens to messages published while a stream’s group is down on this server?

If the stream is replicated and has a leader elsewhere, publishes succeed and this server’s replica catches up when it comes back. If the affected group was the leader, the group elects a new one from its remaining peers and publishes resume after that election. If the stream is R1 on this server, publishes fail until the server is restarted and the asset recovers. JetStream publish acknowledgments make the distinction unambiguous: an unacknowledged publish was not stored.