NATS Invalid Stream Monitor Metadata

JetStream streams carry operator-defined monitoring thresholds as stream metadata under the io.nats.monitor. prefix. This check fires when one of those recognized keys holds a value that cannot be parsed as the type the key requires. A count key holding 200msgs, a duration key holding 30, a boolean key holding yes. The metadata is stored as written, because NATS does not validate it at write time, and the error only surfaces when something tries to evaluate it.

Why this matters

The failure is not that one threshold stops working. It is that all of them do.

Monitor options are extracted from the metadata as a single set. One value that fails to parse rejects the entire set, so every other io.nats.monitor. threshold on that stream stops being evaluated with it. A stream with correct mirror lag, peer activity, and message count thresholds, plus one typo in peer-expect, is a stream with no monitoring at all.

Nothing about the stream looks wrong while this is true. Messages publish and consume normally, replication continues, and the metadata shows up in nats stream info exactly as written. The alerting is what is broken, and alerting that is broken looks identical to alerting that has nothing to report. You find out during the incident the thresholds were meant to catch.

The upstream nats server check stream command surfaces the parse failure as a critical, with the strconv error inline:

1
Critical | could not configure based on metadata: strconv.ParseUint: parsing "abc": invalid syntax

Every other status line that command would have produced is gone, including a message count warning that was firing before the bad key was added. That is the suppression, visible in one command.

Values are validated whether or not monitoring is switched on. Setting io.nats.monitor.enabled to false does not clear this finding, because the option set still has to parse before anything can read the enabled flag out of it.

Common causes

• A unit appended to a count. io.nats.monitor.lag-critical and io.nats.monitor.msgs-warn take a bare integer. 200msgs, 10k, and 1e6 all fail. Duration keys are the only ones that take a unit suffix, and they are the only ones that require it.

• A duration written without a unit, or with a space. io.nats.monitor.seen-critical: 30 means nothing to the parser. Neither does 30 s, which is what a shell command produces when the quoting is wrong. Only bare 0 is accepted without a unit.

• Using a sentinel to disable a check. Operators reach for -1 or an empty string to mean “skip this one”. For the unsigned count keys, -1 is a parse failure rather than a disable. Zero is the disable value for the threshold keys.

• peer-expect set to zero. This key asserts how many replicas the stream is configured to have, so it must be positive. 0 and -1 parse as integers but are rejected as assertions, and they reject the whole option set with them.

• Templated metadata that rendered empty. A Helm value, Terraform variable, or CI substitution that resolves to nothing leaves the key present with an empty string. An empty value is not the same as an absent key, and it fails every type.

• A non-Go boolean spelling. io.nats.monitor.enabled accepts the twelve spellings Go’s boolean parser accepts. yes, on, enabled, and tRue are not among them.

• A copy-paste from a stream whose thresholds had different types. min-sources and subjects-warn are signed integers, lag-critical and msgs-critical are unsigned. A negative value moved from one to the other stops parsing.

How to diagnose

Read the stream’s monitor metadata

Start with what is actually stored, not what you believe you set:

nats stream info ORDERS --json \
  | jq -r '.config.metadata | to_entries[]
           | select(.key | startswith("io.nats.monitor."))
           | "\(.key) = \(.value)"'

Quoting the values makes trailing whitespace and empty strings visible, which is worth doing when nothing looks obviously wrong:

nats stream info ORDERS --json | jq '.config.metadata'

Check each value against its type

Durations accept ns, us, ms, s, m, and h, plus the extended units d, w, M, and y. Extended units truncate to whole hours, so 0.1d is two hours. Note that M is months and m is minutes: 5M parses cleanly and is five months, which is a valid value that will never fire.

Confirm the thresholds have gone quiet

Run the upstream check against the stream and compare it with one you know is configured correctly:

nats server check stream --stream ORDERS --format text

A stream with valid metadata produces a status line per evaluated threshold. A stream with an invalid value produces the could not configure based on metadata critical and nothing else. The JSON form is easier to diff across many streams:

nats server check stream --stream ORDERS --format json | jq '{status, criticals, warnings, ok}'

Sweep every stream in the account

One bad value is rarely alone, because metadata is usually applied by the same tool across a fleet:

nats stream ls --json | jq -r '.[]' | while read -r stream; do
  nats stream info "$stream" --json \
    | jq -r --arg s "$stream" '.config.metadata // {} | to_entries[]
             | select(.key | startswith("io.nats.monitor."))
             | "\($s)\t\(.key)\t\(.value)"'
done

Pipe that through column -t and scan the value column. Type errors are obvious once the values are lined up.

Find where the value came from

The stored value is a symptom. The generator is the cause:

grep -rn "io.nats.monitor" --include="*.yaml" --include="*.tf" --include="*.json" .

Check the surrounding template for numeric values that pass through a formatter, and for durations built by concatenating a number with a unit variable.

How to fix it

Immediate: correct the value

nats stream edit --metadata replaces the entire metadata map rather than merging into it, so any key you omit is deleted. Dump the current set first and re-apply it in full with the correction:

nats stream info ORDERS --json \
  | jq -r '.config.metadata | to_entries[]
           | select(.key | startswith("_nats.") | not)
           | "--metadata=\(.key)=\(.value)"'

The _nats.* keys are server-owned and are re-added automatically. Take the printed arguments, fix the offending one, and apply them together:

nats stream edit ORDERS \
  --metadata=io.nats.monitor.enabled=true \
  --metadata=io.nats.monitor.lag-critical=200 \
  --metadata=io.nats.monitor.seen-critical=5m \
  --metadata=team=payments \
  --force

Verify the result before moving on, both that the value is right and that nothing else was dropped:

nats stream info ORDERS --json | jq '.config.metadata'
nats server check stream --stream ORDERS --format text

Expect new findings once the option set parses again. The thresholds that were suppressed start evaluating at the next scrape, and any real breach among them reports for the first time. Those are pre-existing conditions surfacing, not new problems introduced by the fix.

Short-term: remove what you cannot classify

If you inherited a value and cannot determine what it was meant to be, remove the key instead of guessing at a threshold. Omit it from the re-applied set. An absent key is ignored, and the remaining thresholds resume working.

If a threshold needs to be off, set it to 0 rather than removing it or using a sentinel. Zero is the documented disable value for the count and duration keys, it parses, and it keeps the intent visible to whoever reads the metadata next. The exception is peer-expect, where zero is invalid: remove that key to stop asserting a replica count.

Turning io.nats.monitor.enabled to false is not a workaround. The values are still parsed, the finding still fires, and you now have no monitoring and an open finding.

Long-term: validate at the point of authorship

Type the values in the source. Where metadata is generated, keep thresholds as typed fields rather than free-form strings, and render them with a formatter that cannot emit a unit or a separator. A count that is an integer in the template cannot become 1,000 in the stream.

Check it in CI. Apply the stream configuration to a scratch server and run the upstream check against it:

nats server check stream --stream ORDERS --format json | jq -e '.status != "CRITICAL"'

A parse failure fails the pipeline instead of reaching production and going quiet.

Keep one owner per stream’s metadata. Because edits replace the whole map, two tools writing metadata to the same stream will overwrite each other’s keys. Decide which one owns it, and have the other read.

Alert on the finding itself. This check is the only signal that the thresholds have stopped evaluating. Route stream-034 to the same place the thresholds it protects would have gone, and treat it with the same urgency, because it means those thresholds are not there.

Frequently asked questions

Which values are invalid, and which merely disable a check?

For the count and duration keys, zero disables the threshold and is a valid value. io.nats.monitor.lag-critical: 0 means “do not check mirror lag” and parses correctly. Invalid means the value cannot be parsed as its type at all: letters in a number, a duration without a unit, a negative number in an unsigned key. The one key where zero is invalid is peer-expect, which asserts a configured replica count and must be positive.

Does disabling monitoring clear this finding?

No. Validation runs regardless of io.nats.monitor.enabled, because the enabled flag lives inside the same option set that has to parse first. Correct or remove the bad value. If you also want monitoring off, set enabled to false after the rest of the metadata parses.

Is a misspelled key name reported?

No. Only recognized keys are validated. io.nats.monitor.msgs-warning is not a recognized key, so it is ignored entirely, which means it never produces a finding and never applies a threshold. If a threshold you configured has never fired, check the key spelling against the table above before assuming the stream is healthy.

Does invalid metadata affect the stream’s data or its clients?

No. Stream metadata is inert to the data path. Publishes, consumes, replication, mirroring, and sourcing are unaffected. The only consequence is that the monitoring thresholds attached to that stream stop being evaluated, by this check and by nats server check stream alike.

Why do other findings appear right after I fix this?

Because they were suppressed. Mirror lag, peer lag, peer activity, source counts, message counts, and subject counts are all evaluated from the same option set the invalid value was rejecting. When the metadata parses, those checks run against the stream’s current state and report whatever they find. Fix the metadata during a window where you can triage what comes back.