NATS JetStream Store Write Error

A stream’s message store on this server has stopped accepting writes, or it failed to come up from disk at all. The stream exists in the cluster and the server is assigned to hold a copy of it, but this copy cannot persist another message. nats-server records the failure the first time a write is refused and keeps returning it until the stream is restarted on that server.

Why this matters

The write error is latched. nats-server stores the first error the message store returns and every subsequent write is rejected with that same stored error, without touching the disk again. Fixing the underlying disk problem does not un-latch it. Until the stream restarts on this server, this replica stays dead no matter how much space you free.

For a replicated stream, nats-server reacts immediately. When the write error is recorded, the stream’s Raft node steps down and is put into observer mode, so it neither leads nor campaigns for leadership. This is deliberate: it hands the stream to a replica that can still write. The stream stays available, but it is now running one replica short, with no fault tolerance left if it was an R3.

For an R1 stream, there is no replica to hand it to. Every publish to that stream fails on this server, and it stays failing until the store is repaired.

The other half of this family is a stream that could not be recovered. nats-server walks the JetStream store directory and expects every stream directory it finds to correspond to a stream it can look up. When it cannot, the stream’s data is on disk but the stream is not running: the store failed to load. The messages are not lost, but nothing is serving them from this server.

The cause matters more than usual here, because the store is reporting an error from the operating system and that error is carried verbatim in the health check text. A no space left on device and a read-only file system need completely different responses.

Common causes

• Disk full or over quota. The volume backing the JetStream store directory reached zero free bytes and the write returned ENOSPC, or a user or project quota returned EDQUOT while the volume still had space. This is the most common cause by a wide margin, and it is usually not the stream that filled the disk. Logs, core dumps, and a co-tenant process sharing the volume are frequent culprits.

• Inode exhaustion. The volume has free bytes but no free inodes, so creating a new message block fails with ENOSPC even though df -h looks healthy. JetStream creates many small files per stream, which makes this more likely than on a typical workload.

• Filesystem remounted read-only. Linux remounts a filesystem read-only after certain I/O errors, and some volume drivers do the same when a backing device becomes unavailable. Writes fail with EROFS and the store latches immediately.

• Permissions changed on the store directory. A configuration management run, a container image change, or a chown during a migration leaves the directory owned by a user the nats-server process is not. Writes fail with EACCES. This also happens after a restore that did not preserve ownership.

• Underlying device errors. A failing disk or a network volume that dropped its connection returns EIO. Check the kernel log as well as the NATS log, because the device error is reported there first.

• Store that failed to load at startup. A stream directory present on disk that nats-server could not recover into a running stream produces the could not be recovered cause. Truncated or corrupt state files from an unclean shutdown, a partial restore, or a volume attached at the wrong point are the usual reasons.

How to diagnose

Read the error the server latched

The check reads /healthz with details enabled so every failing stream is listed rather than only the first:

curl -s 'http://localhost:8222/healthz?details=true' | jq

Entries in this family carry the operating system’s error in the text:

1
{
2
  "status": "error",
3
  "errors": [
4
    {
5
      "type": "STREAM",
6
      "account": "ORDERS",
7
      "stream": "EVENTS",
8
      "error": "JetStream stream 'ORDERS > EVENTS' is not current: stream write error: write /data/jetstream/ORDERS/streams/EVENTS/msgs/42.blk: no space left on device"
9
    }
10
  ]
11
}

A stream that failed to load reports without an operating system error, as JetStream stream 'ORDERS > ARCHIVE' could not be recovered.

Read the same data across the fleet, or from one server:

nats server request jetstream-health --details
nats server request jetstream-health --details --name nats-3

The server log carries the same failure at the moment it happened, which is the one timestamp worth having:

1
[ERR] JetStream stream 'ORDERS > EVENTS' critical write error: write ...: no space left on device

Check space and inodes on the store directory

Find the directory the server is actually using rather than assuming the configured path:

STORE=$(curl -s http://localhost:8222/varz | jq -r '.jetstream.config.store_dir')
df -h "$STORE"
df -i "$STORE"

Check both. A volume at 100% inodes with 40% bytes free fails writes while looking fine in every dashboard that only graphs bytes.

Check permissions and mount flags

findmnt -T "$STORE" -o TARGET,SOURCE,FSTYPE,OPTIONS
stat -c '%U:%G %a %n' "$STORE"
ps -o user= -p "$(pgrep -x nats-server)"

ro in the mount options confirms a read-only remount. A directory owner that does not match the nats-server process user confirms a permissions change. Confirm the process can actually write, and check the kernel log for the device error that preceded a remount:

sudo -u nats touch "$STORE/.write-probe" && sudo -u nats rm "$STORE/.write-probe"
dmesg -T | tail -50

Confirm a replica elsewhere is leading the stream

The replicated stream has already stepped down here. Before repairing this server, confirm that the handover worked:

nats stream info EVENTS --json | jq '{leader: .cluster.leader, replicas: [.cluster.replicas[] | {name, current, offline, lag}]}'

A leader on another server with the remaining replicas current means the stream is serving and you are restoring redundancy. No leader means the stream has lost quorum and that is the more urgent problem.

Check whether other streams on this server are affected too, since a full or read-only volume rarely stops at one. A full volume also fails the Raft write-ahead logs, reported separately by Raft WAL Write Error (server-064), so read the groups as well:

nats server report jetstream --host nats-3
curl -s http://localhost:8222/jsz | jq '{storage, reserved_storage, streams, max_storage: .config.max_storage}'
nats server request raft --account ORDERS

How to fix it

Immediate: restore writability and confirm the handover

Free space, or make the filesystem writable again. Remove what filled the volume. Do not delete anything under the JetStream store directory by hand: removing stream data without removing the assignment produces a different failure, reported as JetStream Asset Missing (server-054).

df -h "$STORE"

If the filesystem was remounted read-only, resolve the device error before remounting. Remounting read-write over a failing device repeats the failure within minutes.

Correct ownership if that was the cause:

sudo chown -R nats:nats "$STORE"

Verify the stream is being served from somewhere. A replicated stream should already have a leader on another server:

nats stream info EVENTS --json | jq '.cluster.leader'

Short-term: clear the latched error

The stored write error is never reset while the stream is running. Restarting nats-server on the affected server reloads the stream and clears it:

sudo systemctl restart nats-server

Before restarting, make sure the rest of the cluster can absorb it. Confirm every other stream on this server has a current replica elsewhere, and do not restart a second server until this one is fully caught up:

nats stream report
nats server request jetstream-health --details --name nats-3

If the store is damaged rather than full, do not try to repair it in place. Remove this server from the stream’s group and let the cluster build a clean replica:

nats stream cluster peer-remove EVENTS nats-3
nats stream edit EVENTS --replicas 3

Watch the new replica catch up:

nats stream info EVENTS --json | jq '.cluster.replicas[] | {name, current, lag}'

For an R1 stream there is no replica to rebuild from. A could not be recovered store still holds the data on disk, so take a copy of the stream directory before doing anything else, then restart the server and see whether it loads.

Long-term: keep the store volume from running out

Give JetStream its own volume. Sharing a volume with logs, core dumps, or another service means an unrelated process can take JetStream’s streams down. A dedicated volume turns a shared-disk incident into someone else’s problem.

Set max_storage below the volume size. JetStream then refuses new messages with a stream error before the filesystem refuses the write, which is recoverable. JetStream Storage vs Configured Limit (server-019) reports when the configured limit and the real volume have drifted apart.

Alert on free inodes as well as free bytes. Both fail the same way and only one of them is usually graphed. Alert with enough headroom to act, since the failure mode is a latched error and a restart, not a gradual slowdown.

Set stream limits. A stream with no max_bytes or max_age grows until the disk stops it. Bounding the streams is what keeps the volume-level limits from ever being reached.

Frequently asked questions

I freed up disk space. Why is the stream still failing?

Because the error is latched in memory. nats-server records the first write error the store returns and rejects every later write with that stored error without going back to the disk. Freeing space is necessary but not sufficient. The stream has to be restarted on that server, which in practice means restarting nats-server.

Were messages lost?

Not on this server’s copy of what was already written. The write that failed was rejected, so it was never acknowledged: a JetStream publisher that did not receive an ack knows the message was not stored. For a replicated stream, the write succeeded elsewhere and the surviving replicas hold it. For an R1 stream, the publisher got an error and it is the publisher’s job to retry.

Why did the stream leader move on its own?

That is nats-server protecting the stream. When the write error is recorded on a replicated stream, the Raft node steps down and is placed in observer mode, so it stops leading and stops campaigning. A replica that can still write takes over. It is the reason the stream stayed available rather than stalling on a server that cannot persist anything.

How is this different from Raft WAL Write Error?

This check covers the stream’s own message store. A Raft node’s write-ahead log failing is a separate failure with a separate consequence: the node shuts down, so every asset in that group is unavailable on the server. Raft WAL Write Error (server-064) reports that one, and it names the group and account. The two share root causes on disk and often fire together.

What does “could not be recovered” mean?

It means nats-server found the stream’s directory in the JetStream store but could not load it into a running stream. The data is still on disk. The store did not open, usually because state files were truncated by an unclean shutdown, restored partially, or written by a different server. Take a copy of the directory before attempting anything, then rebuild the replica from the cluster if the stream is replicated.