NATS Pinned Consumer Policy Mismatch: Detecting Misconfigured Pinned Consumers

A pinned consumer policy mismatch alert fires when a JetStream consumer has the io.nats.monitor.pinned metadata key set but its priority_policy is not configured to overflow. This indicates that the operator intends the consumer to behave as a pinned consumer — where one subscriber is the designated primary and others serve as overflow backups — but the consumer configuration doesn’t enforce this behavior. Without the overflow priority policy, all subscribers receive messages via standard distribution, defeating the purpose of the pinned designation.

Why this matters

Pinned consumers are a specialized pattern for workloads that require ordered, single-subscriber processing with automatic failover. The pattern works by designating one subscriber as the “pinned” primary that receives all messages, while additional subscribers wait in standby. If the primary disconnects or becomes unhealthy, an overflow subscriber takes over immediately. This ensures processing continues without interruption while maintaining the ordering guarantees that come from single-subscriber delivery.

The overflow priority policy is the mechanism that enforces this behavior. Without it, the consumer distributes messages across all subscribers using round-robin or other standard distribution logic. This means:

• Ordering is lost. Messages that should be processed sequentially by a single subscriber are instead distributed across multiple subscribers, arriving out of order.
• State partitioning breaks. If the primary subscriber maintains in-memory state that depends on seeing every message (session state, aggregation windows, sequence tracking), distributing messages to other subscribers corrupts that state.
• Failover doesn’t happen correctly. Without the overflow policy, there’s no concept of “primary” and “standby” — all subscribers are peers. The automatic failover that pinned consumers are designed for doesn’t exist.

The metadata annotation signals operator intent: “this consumer should be pinned.” The priority policy enforces it at the infrastructure level. When these two are out of sync, the system behaves differently from what the operator expects, which can lead to subtle data processing errors that are difficult to diagnose after the fact.

This mismatch often occurs during consumer migration or when teams copy consumer configurations between environments without preserving all settings. The metadata gets carried over but the priority policy — which may require a specific server version or configuration syntax — gets dropped.

Common causes

• Consumer created before priority_policy support. The operator added the io.nats.monitor.pinned metadata to mark their intent, but the NATS server version at the time didn’t support priority_policy. After upgrading the server, the consumer configuration was never updated to add the policy.

• Configuration template missing the priority_policy field. Infrastructure-as-code templates (Terraform, Helm, Pulumi) define the consumer with metadata but omit the priority_policy field. Every consumer created from the template has the mismatch.

• Consumer edited and policy dropped. When updating a consumer configuration (changing ack_wait, filter subject, etc.), the priority_policy was accidentally removed. Some CLI or API workflows require specifying the full configuration on update, and omitting a field resets it to the default.

• Manual consumer creation with incomplete configuration. An operator creates a consumer via the CLI or API and adds the metadata but forgets to set priority_policy=overflow. The metadata is documentation-level; it doesn’t automatically configure the priority behavior.

• Environment-specific configuration drift. The consumer is correctly configured in production but a staging or development environment has a copy that lacks the priority policy. The metadata matches across environments but the behavior differs.

How to diagnose

Check consumer configuration

nats consumer info STREAM_NAME CONSUMER_NAME -j | \
  jq '{metadata: .config.metadata, priority_policy: .config.priority_policy}'

If the output shows io.nats.monitor.pinned in metadata but priority_policy is null or absent, the mismatch is confirmed.

List all consumers with pinned metadata

# Find all consumers across streams that have pinned metadata
for stream in $(nats stream list -n); do
  for consumer in $(nats consumer list "$stream" -n); do
    result=$(nats consumer info "$stream" "$consumer" -j 2>/dev/null | \
      jq -r 'select(.config.metadata["io.nats.monitor.pinned"] != null) |
        "\(.stream_name)/\(.name): priority_policy=\(.config.priority_policy // "NOT SET")"')
    [ -n "$result" ] && echo "$result"
  done
done

This shows all consumers marked as pinned and whether their priority policy is correctly configured.

Verify server version supports priority_policy

nats server report jetstream

The priority_policy feature requires NATS Server 2.11 or later. If your servers are on an older version, the policy cannot be set even if the metadata is present.

Programmatic detection

1
// Go: detect pinned metadata without overflow policy.
2
// PriorityPolicy lives in github.com/nats-io/nats.go/jetstream — the
3
// legacy nats.JetStreamContext API does not expose it.
4
import (
5
    "context"
6
    "fmt"
7

8
    "github.com/nats-io/nats.go"
9
    "github.com/nats-io/nats.go/jetstream"
10
)
11

12
ctx := context.Background()
13
js, _ := jetstream.New(nc)
14
stream, _ := js.Stream(ctx, "STREAM_NAME")
15

16
consumers := stream.ListConsumers(ctx)
17
for ci := range consumers.Info() {
18
    _, hasPinned := ci.Config.Metadata["io.nats.monitor.pinned"]
19
    if hasPinned && ci.Config.PriorityPolicy != jetstream.PriorityPolicyOverflow {
20
        fmt.Printf("MISMATCH: %s/%s has pinned metadata but priority_policy=%v\n",
21
            ci.Stream, ci.Name, ci.Config.PriorityPolicy)
22
    }
23
}

1
# Python: detect pinned consumer policy mismatch
2
import nats
3

4
nc = await nats.connect()
5
js = nc.jetstream()
6

7
consumers = await js.consumers_info("STREAM_NAME")
8
for ci in consumers:
9
    metadata = ci.config.metadata or {}
10
    if "io.nats.monitor.pinned" in metadata:
11
        policy = getattr(ci.config, "priority_policy", None)
12
        if policy != "overflow":
13
            print(f"MISMATCH: {ci.stream_name}/{ci.name} "
14
                  f"has pinned metadata but priority_policy={policy}")

How to fix it

Update the consumer to use overflow priority policy

The fix is to add (or correct) the priority_policy on the consumer configuration:

nats consumer edit STREAM_NAME CONSUMER_NAME --priority-policy overflow

If the consumer needs to also specify a priority group (to coordinate across multiple consumers), set the group as well:

nats consumer edit STREAM_NAME CONSUMER_NAME \
  --priority-policy overflow \
  --priority-groups "pinned-group-1"

Verify the fix

After updating, confirm the configuration is correct:

nats consumer info STREAM_NAME CONSUMER_NAME -j | \
  jq '{metadata: .config.metadata, priority_policy: .config.priority_policy, priority_groups: .config.priority_groups}'

Both io.nats.monitor.pinned metadata and priority_policy: "overflow" should be present.

Update infrastructure-as-code templates

If consumers are managed via Terraform, Helm, or other IaC tools, update the template to include the priority policy:

1
# Terraform example
2
resource "jetstream_consumer" "pinned_processor" {
3
  stream_id = jetstream_stream.orders.id
4

5
  durable_name    = "pinned-processor"
6
  ack_policy      = "explicit"
7
  priority_policy = "overflow"
8

9
  metadata = {
10
    "io.nats.monitor.pinned" = "true"
11
  }
12
}

If the server version doesn’t support priority_policy

Upgrade to NATS Server 2.11 or later. If an immediate upgrade isn’t possible, remove the io.nats.monitor.pinned metadata to suppress the mismatch alert — but be aware that the consumer won’t behave as a pinned consumer until the policy can be applied:

# Remove pinned metadata if overflow policy isn't available yet
nats consumer edit STREAM_NAME CONSUMER_NAME \
  --metadata "io.nats.monitor.pinned="

Prevent future mismatches

Add validation to your consumer creation workflow. Any process that sets io.nats.monitor.pinned metadata should also set priority_policy=overflow. This can be enforced via CI checks on IaC templates or wrapper scripts around consumer creation.

Use Synadia Insights for ongoing drift detection. Insights evaluates this check across all consumers every collection epoch, catching mismatches introduced by manual changes, failed deployments, or configuration drift between environments.

Frequently asked questions

What does the overflow priority policy actually do?

The overflow priority policy ensures that messages are delivered to the highest-priority subscriber first. If that subscriber is unavailable (disconnected or slow), messages “overflow” to the next subscriber in priority order. This creates a primary/standby pattern where one subscriber handles all messages under normal conditions, and backups take over automatically during failures.

Can I use pinned consumers with pull-based consumers?

Yes. Pinned consumers with the overflow priority policy work with pull-based consumers. The priority determines which subscriber’s fetch requests are fulfilled first. If the primary subscriber stops fetching, the overflow subscriber’s pending fetch requests receive messages instead.

What happens if I have pinned metadata but no subscribers?

The consumer exists but no messages are delivered (no subscribers to deliver to). The pinned metadata and priority policy only affect delivery behavior when subscribers are present. The consumer simply accumulates pending messages in the stream.

Is the pinned metadata key required for overflow priority policy?

No. You can set priority_policy=overflow without the io.nats.monitor.pinned metadata. The metadata is used by monitoring tools like Synadia Insights to verify that the operator’s intent (pinned behavior) matches the configuration (overflow policy). Setting the policy without the metadata is valid but means monitoring tools won’t flag the consumer for this specific consistency check.

What priority groups should I use?

Priority groups are used when multiple consumers on the same stream coordinate their pinned behavior. Subscribers within the same priority group share the overflow failover logic. If you have a single pinned consumer, a single group suffices. For complex topologies with multiple pinned consumers handling different subject partitions, use distinct group names to keep their failover logic independent.

Does changing priority_policy disrupt active subscribers?

The consumer configuration update may briefly interrupt delivery as the consumer’s internal state is updated. Active subscribers will reconnect and resume. Plan the change during a low-traffic period if the consumer processes latency-sensitive workloads, though the disruption is typically subsecond.