NATS Max Sources: Detecting Unexpected Stream Sources

A JetStream stream that aggregates from multiple source streams can grow its source list beyond what operators intended. When the number of configured sources exceeds the operator-defined maximum — set via the io.nats.monitor.max-sources metadata tag — this check fires. An unexpected source means data is flowing into the aggregate stream from an origin that was not planned for, which can indicate misconfiguration, unauthorized cross-account replication, or runaway automation that is provisioning sources without proper governance.

Why this matters

Every source added to a stream creates a replication pipeline. The aggregate stream’s server must maintain a consumer on each source stream, track replication offsets, and apply any subject transforms. Each source consumes CPU, memory, and network bandwidth on both the source and destination servers. Adding sources beyond the intended set has cascading operational consequences.

First, there is the storage impact. Each source contributes messages to the aggregate stream. An unexpected source may inject a high volume of messages that was not accounted for in the stream’s storage limits, pushing it toward its byte or message count limits faster than planned. If the aggregate stream has no limits configured, the unexpected source can cause unbounded storage growth.

Second, there is the security and compliance dimension. In multi-tenant NATS deployments, streams may source data from other accounts via exports and imports. An unauthorized source means data from an unexpected origin is being mixed into the aggregate. This could violate data isolation requirements, compliance boundaries, or introduce untrusted data into a trusted pipeline.

Third, there is the operational complexity. More sources mean more replication relationships to monitor, more potential points of failure, and more surface area for lag and staleness issues. Operators who expect a known set of sources will be surprised by unexpected entries in the source list, making troubleshooting harder during incidents.

The max-sources check serves as a guardrail — it ensures the source list does not grow beyond what was deliberately planned.

Common causes

• Automation provisioned extra sources. A CI/CD pipeline, Terraform module, or custom operator added sources to the stream without proper bounds checking. This is common when source provisioning is driven by service registration — each new service instance adds itself as a source, and no one enforced a maximum.

• Manual misconfiguration. An operator added a source to the wrong aggregate stream. In environments with multiple aggregate streams (e.g., one per region or per domain), it is easy to accidentally target the wrong stream during a nats stream edit command.

• Cross-account source added without authorization. In multi-account deployments, a new export/import pair was configured that allows a previously unconnected account to source into the aggregate stream. The stream configuration was updated to include this new source, but the change was not reviewed or approved.

• Duplicate sources with different configurations. The same logical source stream was added twice — once with a subject transform and once without, or with different starting sequence numbers. The stream now has two source entries pointing at the same upstream stream.

• Test or development sources left in production. A source was added during testing or debugging and never removed. It may be contributing minimal traffic, making it easy to overlook.

How to diagnose

Check the current source count and threshold

# View source count
nats stream info AGGREGATE_STREAM --json | jq '.config.sources | length'

# View the max-sources threshold
nats stream info AGGREGATE_STREAM --json | jq '.config.metadata["io.nats.monitor.max-sources"]'

List all configured sources

nats stream info AGGREGATE_STREAM --json | jq -r '.config.sources[] | "\(.name) (filter: \(.filter_subject // "none"))"'

Review each entry against your expected source list. Identify which sources are unexpected.

Check for duplicate sources

nats stream info AGGREGATE_STREAM --json | jq '[.config.sources[].name] | group_by(.) | map(select(length > 1)) | .[][0]'

If this returns any stream names, you have duplicate source entries.

Inspect source activity

nats stream info AGGREGATE_STREAM --json | jq '.sources[] | {name: .name, lag: .lag, active: .active}'

Unexpected sources that show recent activity are actively injecting data into the aggregate stream. Sources with no activity may be stale leftovers.

Programmatic detection in Go

1
js, _ := nc.JetStream()
2
info, _ := js.StreamInfo("AGGREGATE_STREAM")
3

4
maxSources := 5 // your expected maximum
5
actualSources := len(info.Config.Sources)
6

7
if actualSources > maxSources {
8
    log.Printf("ALERT: stream has %d sources, expected at most %d",
9
        actualSources, maxSources)
10
    for _, src := range info.Config.Sources {
11
        log.Printf("  source: %s (filter: %s)",
12
            src.Name, src.FilterSubject)
13
    }
14
}

Programmatic detection in Python

How to fix it

Immediate: remove unauthorized or accidental sources

Identify the unexpected source and remove it from the stream configuration. You cannot remove a single source via the CLI in one command — you need to edit the stream configuration:

# Export current config
nats stream info AGGREGATE_STREAM --json | jq '.config' > stream-config.json

# Edit stream-config.json to remove the unwanted source from the "sources" array
# Then apply:
nats stream update AGGREGATE_STREAM --config stream-config.json

If the unexpected source injected unwanted messages, you may need to purge those messages from the aggregate stream. If the source used a distinct subject prefix, you can purge selectively:

nats stream purge AGGREGATE_STREAM --subject "events.unwanted.>"

Short-term: audit all aggregate streams

Identify all streams using sources and check for unexpected entries:

for stream in $(nats stream ls -n); do
  sources=$(nats stream info "$stream" --json 2>/dev/null | jq '.config.sources | length')
  if [ "$sources" -gt 0 ]; then
    echo "$stream: $sources sources"
    nats stream info "$stream" --json | jq -r '.config.sources[].name' | sed 's/^/  /'
  fi
done

Long-term: enforce source governance

Set the io.nats.monitor.max-sources metadata tag. This enables automated monitoring to detect source-count increases:

nats stream edit AGGREGATE_STREAM \
  --metadata "io.nats.monitor.max-sources=5"

Use infrastructure-as-code for stream configuration. Define all stream configs — including the complete sources list — in version-controlled files. Require code review for changes. This prevents ad-hoc source additions.

Restrict stream edit permissions. Use NATS account-level permissions to control who can modify stream configurations. In multi-tenant deployments, use the operator/account model to prevent accounts from modifying streams they do not own.

Implement pre-deployment validation. In your CI/CD pipeline, add a validation step that checks the source count against the max-sources threshold before applying stream configuration changes. Reject deployments that would exceed the limit.

Combine with min-sources for exact enforcement. If you expect exactly N sources, set both io.nats.monitor.min-sources and io.nats.monitor.max-sources to N. Any deviation — addition or removal — triggers an alert.

Frequently asked questions

Can too many sources cause performance problems?

Yes. Each source creates a replication pipeline that consumes CPU, memory, and network bandwidth. The aggregate stream’s server must maintain an internal consumer for each source, process incoming messages, and apply any subject transforms. At scale — dozens or hundreds of sources on a single stream — this can create noticeable resource pressure. High source counts also increase the complexity of stream info responses, which affects monitoring tools that poll stream state frequently.

What if sources are added dynamically by design?

Some architectures intentionally add sources dynamically — for example, when new microservices register themselves and their event streams are automatically sourced into a central aggregate. In this case, set max-sources to a reasonable upper bound rather than an exact count. The check then serves as a circuit breaker: if the source count exceeds the expected ceiling, something unexpected is happening (a provisioning bug, a runaway loop, or an environmental misconfiguration).

How do I find which source was added most recently?

The NATS stream info API does not record when individual sources were added. However, you can infer the most recently added source by checking the active and lag fields in the stream info response. A source with very high lag relative to others was likely added recently and is still catching up. For definitive answers, check your infrastructure-as-code change history or NATS server logs for stream update events.

Is there a hard limit on the number of sources a stream can have?

NATS does not impose a hard limit on source count. The practical limit depends on available server resources and the aggregate message rate from all sources combined. However, streams with very high source counts (50+) are unusual and often indicate an architectural concern — consider whether a different aggregation pattern (such as subject-based routing into a single stream without sources) would be more appropriate.

Does removing a source delete its messages from the aggregate stream?

No. Messages already replicated into the aggregate stream remain there. Removing a source only stops future replication. If you need to remove messages that originated from a specific source, use nats stream purge with a subject filter matching that source’s messages.