NATS JetStream Asset Missing

The JetStream meta layer holds an assignment that names this server as a member of a stream’s or consumer’s Raft group. The server looked for that asset locally and did not find it. The assignment says the asset belongs here; the server says it is not here. Those two facts disagree, and the server reports the disagreement through /healthz.

Why this matters

This is not a rejected configuration. When the cluster refuses a stream or consumer config, the asset never existed and never will until the config changes, which is what Stream Recovery Failure (server-015) reports. Here the asset is supposed to be present. Something removed it, never delivered it, or left the Raft group half-built.

The immediate consequence is reduced redundancy. An R3 stream with one server missing its replica is running as an effective R2: it still has quorum and still serves traffic, but it has no remaining fault tolerance. The next server to go down takes the stream offline. Nothing in the client-facing API surfaces this. Publishes succeed, consumers make progress, and nats stream info reports the configured replica count, so the degraded state stays invisible until the second failure turns it into an outage.

For an R1 stream, there is no redundancy to lose. The asset is gone from the only server that was supposed to hold it, and every operation against it fails.

The other direction matters just as much. If the asset exists nowhere in the cluster, the assignment on this server is a leftover from a delete that propagated to some peers and not others. The meta layer is carrying state for an asset that no longer exists, and the server keeps failing its health check over an asset nobody wants back.

Common causes

• Stuck Raft catch-up. A replica was added to the group but never completed its initial catch-up. The stream directory on this server is empty or partial, so lookupStream fails while the assignment persists. A catch-up stalls when the leader is under load, when the snapshot transfer is interrupted, or when the follower runs out of disk mid-transfer.

• Placement change that did not complete. A peer-remove followed by a scale-up, a rebalance, or an edit --replicas moves assignments between servers. If the new member never came up and the old assignment was not cleaned, both sides can end up inconsistent.

• Delete that propagated partially. A stream or consumer was deleted while one server was partitioned, restarting, or otherwise not applying meta entries. The delete landed everywhere else. This server kept the assignment.

• Store directory removed underneath the server. The JetStream store directory, or an account or stream subdirectory inside it, was deleted, moved, or restored from a backup that predates the asset. The server starts, reads the assignment from the meta layer, and finds nothing on disk.

• Consumer created against a stream that is not up here. isConsumerHealthy reports stream missing when the parent stream is not present on the server. One missing stream produces a missing finding for every consumer it owns, so a burst of consumer findings usually points at a single stream.

• Raft group node never initialized. The assignment exists and the asset exists, but the group’s node handle is absent (group node missing, stream node missing, consumer node missing). This shows up during a restart window or after a failed group creation, and it only applies to replicated assets, since R1 assets skip the group checks entirely.

How to diagnose

Read the healthz errors on the affected server

The check reads /healthz with details enabled, which returns every failing asset instead of stopping at the first one:

curl -s 'http://localhost:8222/healthz?details=true' | jq

With details=true the response has "status": "error" and an errors array. Each entry names the asset and carries the cause:

1
{
2
  "status": "error",
3
  "errors": [
4
    {
5
      "type": "STREAM",
6
      "account": "ORDERS",
7
      "stream": "EVENTS",
8
      "error": "JetStream stream 'ORDERS > EVENTS' is not current: stream not found"
9
    },
10
    {
11
      "type": "CONSUMER",
12
      "account": "ORDERS",
13
      "stream": "EVENTS",
14
      "consumer": "fulfillment",
15
      "error": "JetStream consumer 'ORDERS > EVENTS > fulfillment' is not current: stream missing"
16
    }
17
  ]
18
}

The causes in this family are stream not found, consumer not found, stream missing, stream assignment or group missing, consumer assignment or group missing, group node missing, stream node missing and consumer node missing. Without details=true the server returns HTTP 503 with "status": "unavailable" and only the first error.

The same data is available over NATS across the whole fleet, or from one server:

nats server request jetstream-health --details
nats server request jetstream-health --details --name nats-3

Check whether the asset exists elsewhere in the cluster

This is the question that decides which way to fix it:

nats stream info EVENTS
nats consumer info EVENTS fulfillment
nats server request jetstream-health --details --account ORDERS --stream EVENTS

If the stream reports normally, the asset is alive and this server missed it. If both report that the stream or consumer does not exist, the assignment here is a leftover from a partial delete. The scoped health request shows what each server in the fleet says about that one asset.

Compare the assignment against the Raft group peers

nats stream info EVENTS --json | jq '.cluster'

The cluster.replicas array lists each peer with current, offline and lag. A server that reports the asset as missing normally appears here as not current, or does not appear at all while the meta layer still assigns it.

Read the group directly from the server that is failing:

nats server request raft --account ORDERS

Look for the group backing the stream and check whether this server is listed as a peer and what state its node is in.

Check whether a catch-up is stuck

A replica that is behind but recovering is a different condition, owned by Raft Sustained Catching Up (server-048). A replica that never started is what produces a missing finding. Compare the two:

nats stream info EVENTS --json | jq '.cluster.replicas[] | {name, current, lag, active}'

A peer with a large, unchanging lag across two readings is stalled. A peer absent from the list while the meta layer assigns it never got that far.

Check the store directory and the server log

STORE=$(curl -s http://localhost:8222/varz | jq -r '.jetstream.config.store_dir')
ls -l "$STORE/ORDERS/streams"

An account directory with no subdirectory for the stream confirms that the asset never landed on disk. In the server log, look for the group creation and catch-up lines for that stream, and for any error during store recovery at startup.

How to fix it

Immediate: establish whether the stream is still serving

nats stream info EVENTS --json | jq '{leader: .cluster.leader, replicas: [.cluster.replicas[].name]}'

If a leader is elected and the remaining replicas are current, the stream is up and you are working on restoring redundancy, not on an outage. Say so explicitly before anyone starts restarting servers.

If no leader is elected, the stream has lost quorum and that is the problem to solve first.

Short-term: reconcile the assignment

When the asset exists elsewhere, remove this server from the group and let the cluster place a fresh replica. Removing the stale peer also removes the consumer groups that follow the stream:

nats stream cluster peer-remove EVENTS nats-3
nats stream info EVENTS --json | jq '.cluster.replicas | length'

If the replica count dropped, restore it and let placement pick a server:

nats stream edit EVENTS --replicas 3

Watch the new replica catch up before calling it done:

nats stream info EVENTS --json | jq '.cluster.replicas[] | {name, current, lag}'

When the group node is missing rather than the asset, restart the server. A node handle that never initialized does not repair itself, and a restart rebuilds the group from the assignment. Find out what prevented the group from starting before you restart, because a restart erases the evidence.

When the asset exists nowhere, the assignment is a leftover. Recreate the asset and delete it cleanly so the delete propagates to every peer, including this one:

nats stream add EVENTS --config /tmp/events.json
nats stream rm EVENTS

Confirm that the healthz error is gone on every server, not just the one that reported it:

nats server request jetstream-health --details

Long-term: keep assignments and assets from diverging

Do not delete anything under the JetStream store directory by hand. The meta layer is the record of what should exist. Removing state on disk without removing the assignment produces exactly this finding. Use nats stream rm and nats consumer rm, and let the delete replicate.

Give catch-up room to finish during rolling restarts. Wait for every replica to report current: true before restarting the next server. A restart loop that moves faster than catch-up leaves groups half-built:

nats stream report

Reserve headroom on the store volume. A follower that fills its disk during a snapshot transfer abandons the catch-up. JetStream Storage vs Configured Limit (server-019) is the earlier warning for that.

Restore backups with the meta state that matches them. A store directory restored from a point in time that predates a stream will not contain it, while the meta layer restored from a later point will still assign it.

Frequently asked questions

Does the stream still work when one server reports this?

Usually yes, and that is the trap. A replicated stream with one replica missing still has quorum and still serves publishes and consumes. What it has lost is the ability to survive the next failure. The client-facing API does not report the degraded state, which is why the health check is worth acting on before anything else breaks.

Why did a consumer I just created show up as missing?

It should not. nats-server suppresses the consumer not found result for the first five seconds after the consumer assignment was created, so a consumer that is still coming up does not report as missing. If the finding persists past that window, the consumer genuinely did not start on that server.

How is this different from a rejected configuration?

A rejected configuration means the cluster refused the asset, so it does not exist and will not exist until the configuration is corrected. Stream Recovery Failure (server-015) reports that case, and it does not clear on its own. A missing asset is one the cluster accepted and assigned to this server. The configuration is fine; the delivery is not.

Will restarting the server fix it?

It depends on the cause. A missing group node is usually resolved by a restart, because the node is rebuilt from the assignment. A stream directory that is not on disk is not, because a restart reads the same empty directory and reaches the same conclusion. Determine which cause you have from the healthz text before restarting.

Can this fire on a non-clustered server?

No. isStreamHealthy and isConsumerHealthy return early when the server is not running in clustered mode, because there is no meta assignment to disagree with. A single server running JetStream on its own never produces this finding. It can, however, report the store family instead when a stream fails to recover from disk.