NATS Invalid Consumer Monitor Metadata

Consumer monitoring thresholds are stored on the consumer as metadata under the io.nats.monitor. prefix: how many outstanding acks are too many, how long a consumer may go without delivering, whether pinned client priority is required. This check fires when one of those recognized keys holds a value that cannot be parsed as the type its key requires. NATS stores metadata verbatim and does not type-check it on write, so the value sits there looking plausible until something tries to read it.

Why this matters

The consumer thresholds are the ones that page someone. Outstanding acks climbing, a worker pool that has stopped acknowledging, redeliveries running away, a consumer that has delivered nothing in an hour. Those are the conditions you find out about from an alert rather than from a dashboard, which is exactly why it matters that a single bad value switches all of them off.

Monitor options are read as one set. A value that does not parse rejects the entire set, so every threshold on that consumer goes with it. A malformed io.nats.monitor.pinned boolean, which has nothing to do with acknowledgement, takes out outstanding-ack-critical, last-ack-critical, redelivery-critical, and the rest along with the pinning check.

The consumer itself is fine. Messages are delivered, acknowledged, and redelivered normally, and the metadata appears in nats consumer info exactly as it was written. Only the evaluation is broken, and a threshold that is not evaluated is indistinguishable from a threshold that is not breached. The gap stays invisible until the condition it was configured for actually happens.

Consumers make this worse than it is for streams, for two reasons. There are more of them, often hundreds per account, so a generator that emits one bad value emits it many times. And they are usually created by application code rather than by an operator, so the value is compiled into a service that recreates the consumer on every deploy, and a fix applied with the CLI is reverted at the next restart.

Values are validated whether or not monitoring is enabled. Setting io.nats.monitor.enabled to false does not clear this finding, because the enabled flag lives inside the same option set that has to parse first.

Common causes

• A duration expressed as a number. io.nats.monitor.last-ack-critical and last-delivery-critical are durations and need a unit. A client that holds the timeout as milliseconds and serializes it directly writes 30000, which does not parse. Only bare 0 is accepted without a unit.

• A unit the parser does not know. 30sec, 5min, and 2hrs are all rejected. The accepted spellings are ns, us, ms, s, m, and h, plus the extended units d, w, M, and y.

• A space inside a duration. 30 s is what you get from an unquoted shell variable or a formatter that pads its output. The space breaks the component, and the whole value is rejected.

• A boolean written the way the rest of your config writes booleans. io.nats.monitor.enabled and io.nats.monitor.pinned accept the twelve spellings Go’s boolean parser accepts. YAML-style yes, on, and y are not among them, and a YAML file that quotes the value passes the literal string straight through.

• A count with a separator or a decimal. outstanding-ack-critical, waiting-critical, unprocessed-critical, and redelivery-critical take a plain integer. 1,000, 1000.0, and 1e3 all fail, and a language whose default number formatting produces any of those will produce them here.

• A templated value that rendered empty. A Helm value or environment variable that resolves to nothing leaves the key present holding an empty string. An empty value is not an absent key, and it fails every type.

• A consumer cloned from a template with different key types. Copying a block of monitor metadata between consumers carries the values but not the knowledge of which key each one belonged to. A duration landing on a count key stops parsing.

How to diagnose

Read the consumer’s monitor metadata

Work from what is stored rather than from what the deployment manifest says:

nats consumer info ORDERS WORKERS --json \
  | jq -r '.config.metadata | to_entries[]
           | select(.key | startswith("io.nats.monitor."))
           | "\(.key) = \(.value)"'

When nothing looks wrong, print the raw object instead. Quoted values expose trailing whitespace and empty strings, which are the two failures that survive a visual scan:

nats consumer info ORDERS WORKERS --json | jq '.config.metadata'

Check each value against its type

Zero or less disables an integer threshold and is a valid value, so -1 is a working way to switch one off. It is not a way to switch off a duration or a boolean, where it does not parse. Note also that M is months and m is minutes: 5M is valid and means five months, which is a threshold that will never fire.

Sweep every consumer in the account

One malformed value usually means many, because the same code created all of them:

nats stream ls --json | jq -r '.[]' | while read -r stream; do
  nats consumer ls "$stream" --json | jq -r '.[]' | while read -r consumer; do
    nats consumer info "$stream" "$consumer" --json \
      | jq -r --arg s "$stream" --arg c "$consumer" \
          '.config.metadata // {} | to_entries[]
           | select(.key | startswith("io.nats.monitor."))
           | "\($s)/\($c)\t\(.key)\t\(.value)"'
  done
done

Pipe the output through column -t and read down the value column. Type errors stand out once the values are aligned, and the pattern tells you which service emitted them.

Evaluate the threshold out of band

Command-line flags override metadata on the upstream check, so you can run the threshold the consumer was supposed to have and see what it would have reported while the metadata was unparseable:

nats server check consumer --stream ORDERS --consumer WORKERS \
  --outstanding-ack-critical=1000 \
  --last-ack-critical=90s \
  --format text

Do this before you fix the metadata. It tells you whether the suppressed threshold was quietly sitting on a real breach, which decides how urgently the fix has to go out.

Find the code that creates the consumer

A consumer that an application asserts on startup will be rewritten with the same bad value on the next deploy:

grep -rn "io.nats.monitor" --include="*.go" --include="*.ts" --include="*.py" \
  --include="*.java" --include="*.yaml" .

Look for durations built by concatenation, numbers passed through a locale-aware formatter, and booleans that pass through a YAML or JSON config layer before reaching the consumer config.

How to fix it

Immediate: correct the value

nats consumer edit --metadata replaces the whole metadata map rather than merging into it, so any key you leave out is deleted. Dump the current set first:

nats consumer info ORDERS WORKERS --json \
  | jq -r '.config.metadata | to_entries[]
           | select(.key | startswith("_nats.") | not)
           | "--metadata=\(.key)=\(.value)"'

The _nats.* keys are server-owned and come back on their own. Take the printed arguments, correct the offending value, and apply the set in one edit:

nats consumer edit ORDERS WORKERS \
  --metadata=io.nats.monitor.enabled=true \
  --metadata=io.nats.monitor.outstanding-ack-critical=1000 \
  --metadata=io.nats.monitor.last-ack-critical=90s \
  --force

Confirm what is stored afterwards, checking both the corrected value and that the other keys survived:

nats consumer info ORDERS WORKERS --json | jq '.config.metadata'

Expect findings to appear once the set parses. Outstanding acks, waiting pulls, unprocessed messages, redeliveries, delivery and acknowledgement staleness, and the pinned policy check all resume at the next scrape and report whatever the consumer’s current state deserves.

Short-term: fix it where the consumer is created

A CLI edit lasts until the owning service next asserts its consumer configuration. If the consumer is created by application code, an infrastructure module, or a startup migration, change it there and redeploy, then re-check the stored value. Until that happens, treat the CLI fix as a stopgap and expect the finding to return.

If you cannot determine what a value was meant to be, remove the key rather than inventing a threshold. An absent key is ignored and the remaining thresholds work. If a threshold genuinely needs to be off, set the integer keys to 0, which parses and keeps the intent readable, rather than blanking the value.

Turning io.nats.monitor.enabled to false does not help. The values are still parsed, the finding still fires, and you end up with no monitoring and an open finding.

Long-term: make the value unable to be wrong

Keep thresholds typed in the source. Hold durations as a duration type and counts as an integer, and convert once at the point the metadata map is built. A Go time.Duration formats as 1m30s, which parses. A number of milliseconds formats as 90000, which does not.

Build the metadata in one helper. A single function that takes typed arguments and returns the map is the place to get the spellings right, and the place to add a test. Services that each assemble their own strings each get their own chance to be wrong.

Validate before it ships. Test the helper with the same parsers NATS uses, so a bad spelling fails the build rather than the alert. In Go that is strconv.ParseBool for the boolean keys, strconv.Atoi for the counts, and a duration parse for the two duration keys:

1
for key, value := range consumerMonitorMetadata(cfg) {
2
    switch key {
3
    case "io.nats.monitor.enabled", "io.nats.monitor.pinned":
4
        _, err := strconv.ParseBool(value)
5
        require.NoError(t, err, key)
6
    case "io.nats.monitor.last-delivery-critical", "io.nats.monitor.last-ack-critical":
7
        _, err := time.ParseDuration(value)
8
        require.NoError(t, err, key)
9
    default:
10
        _, err := strconv.Atoi(value)
11
        require.NoError(t, err, key)
12
    }
13
}

time.ParseDuration covers the standard units only. If you use the extended d, w, M, or y units, validate with a parser that knows them rather than loosening the test.

Alert on this finding. Consumer-020 is the only indication that a consumer’s thresholds have stopped evaluating. Route it to wherever the thresholds themselves would have gone, because that is the alert you are not going to get.

Frequently asked questions

Which values are invalid, and which only disable a check?

Zero or less disables the integer thresholds and is a valid value: io.nats.monitor.redelivery-critical: 0 means “do not check redeliveries” and parses correctly. A duration of 0 disables the same way. Invalid means the value cannot be parsed at all as the type its key requires: letters in a count, a duration with no unit, a boolean spelled the way YAML spells it.

Does a bad boolean only affect the checks that read it?

No. The option set is extracted as a unit, so a malformed io.nats.monitor.pinned value suppresses acknowledgement, redelivery, waiting, unprocessed, and staleness thresholds along with the pinned policy check. The key that fails to parse has no relationship to the checks that stop running.

The consumer is recreated on every deploy. Is fixing it by CLI worth it?

It restores the thresholds until the next deploy, which is worth doing if the consumer is one you rely on being alerted about. It is not a fix. Change the value in whatever asserts the consumer configuration, ship that, and verify the stored metadata afterwards.

Is a misspelled key name reported?

No. Only recognized keys are validated, so io.nats.monitor.ack-outstanding-critical is ignored entirely: it produces no finding and applies no threshold. If a consumer threshold has never fired, check the spelling against the table above before concluding the consumer has been healthy.

Does this affect message delivery?

No. Consumer metadata is inert to the delivery path. Deliveries, acknowledgements, redeliveries, and pull requests behave exactly as configured. What stops is the evaluation of the monitoring thresholds attached to that consumer.