Debugging CPU spikes in cross-account JetStream stream sourcing

A community member asked why a NATS deployment using multiple accounts and JetStream stream sourcing could suddenly show rising server CPU, memory, and core NATS message rate even though JetStream stream metrics showed very little data traffic.

Short answer

For cross-account JetStream stream sourcing, verify that all three categories of subjects are routed correctly between accounts:

1. The stream data subjects.
2. The JetStream API service subjects used to create the source consumer.
3. The JetStream flow-control subjects, especially $JS.FC.<stream>.>.

A misconfigured flow-control import/export can run quietly under low load and then fail when flow control is actually needed. The result may look like a rapidly increasing core NATS message rate, repeated heartbeat or flow-control traffic, repeated source-consumer activity, and rising CPU or memory, while JetStream message counts remain low.

Also check for subject loops. Cross-account imports and exports can accidentally create loops or ambiguous routes, and loop detection is not something you should rely on as a design mechanism.

Why JetStream metrics may look quiet

JetStream stream metrics primarily help you understand stored stream and consumer activity. In this failure mode, the problem may be in the core NATS control traffic around sourcing: service requests, subscriptions, heartbeats, and flow-control messages.

That can produce an apparent contradiction:

• Server CPU and memory rise.
• Core NATS message rate rises, sometimes very quickly.
• JetStream stream metrics show only a small burst, then little or no stored-message growth.
• Tracing or logs may show repeated idle heartbeat or flow-control related subjects.

This does not rule out a routing problem. It often means the traffic to inspect is not just the application payload subjects stored in the stream.

Checklist: what to inspect first

1. Identify the subject that is multiplying

Use a broad subject report from the account where you suspect the extra traffic is visible:

nats sub '>' --report-subjects

In a multi-account setup, run the check with credentials for each relevant account. A subject may only be visible from one side of an import/export boundary.

On a busy production system, a wildcard subscription can be noisy. If you already have a suspected prefix, narrow the subscription where possible.

2. Check whether source consumers are being created repeatedly

JetStream sourcing creates consumers on the source stream. If routing is wrong, you may see repeated requests around the source-consumer creation API subject, such as:

1
$JS.API.CONSUMER.CREATE.<stream>.>

Look from the account that owns the source stream. If new source-consumer creation requests appear continuously, investigate the corresponding import/export mapping before assuming the problem is with stored messages.

3. Verify flow-control exports and imports

For cross-account stream sourcing, the flow-control subject must be routable. A common trap is exporting or importing the data stream and consumer-create API correctly, but missing or ambiguously mapping the flow-control subject.

The flow-control export should include the stream name, for example:

1
$JS.FC.orders_tenant1.>

Avoid overly broad or ambiguous flow-control mappings when multiple source streams or tenant accounts are involved. The stream name in the flow-control subject is important because it helps avoid routing ambiguity.

4. Keep stream names unambiguous

When sourcing across accounts, do not assume that two streams with the same name are safely distinguishable just because they live in different accounts. In practice, stream names used in a sourcing topology should be unambiguous.

If you need to source from multiple accounts, prefer account- or tenant-specific stream names such as:

1
orders_tenant1
2
orders_tenant2

rather than using the same stream name in each account and relying on account boundaries to disambiguate routing.

If you must integrate existing streams with overlapping names, review the current JetStream sourcing constraints and design explicit subject mappings before deploying the topology.

Example cross-account shape

The exact syntax depends on whether you manage accounts in server configuration, an operator model, or decentralized JWTs, but the routing shape is the same: the source account exports the required stream, consumer-create service, and flow-control service; the destination account imports them.

A simplified example:

1
accounts: {
2
  APP_TENANT1: {
3
    jetstream: enabled
4
    exports: [
5
      { service: "$JS.API.CONSUMER.CREATE.orders_tenant1.>", accounts: ["APP_MAIN"] }
6
      { service: "$JS.FC.orders_tenant1.>", accounts: ["APP_MAIN"] }
7
      { stream: "orders_tenant1.S.>", accounts: ["APP_MAIN"] }
8
    ]
9
  }
10

11
  APP_MAIN: {
12
    jetstream: enabled
13
    imports: [
14
      { service: { account: APP_TENANT1, subject: "$JS.API.CONSUMER.CREATE.orders_tenant1.>" }, to: "$JS.TENANT1.API.CONSUMER.CREATE.orders_tenant1.>" }
15
      { service: { account: APP_TENANT1, subject: "$JS.FC.orders_tenant1.>" } }
16
      { stream: { account: APP_TENANT1, subject: "orders_tenant1.S.>" } }
17
    ]
18
  }
19
}

For multiple source accounts, repeat the pattern with unique stream names and, where needed, account-specific API prefixes. The important point is that the flow-control route should be just as intentional as the data route.

Why this can fail only under load

A flow-control routing problem may not appear immediately. With low message volume, sourcing can seem healthy because the system may not need to exercise flow control heavily.

A burst of messages can change that. Once flow control is needed, missing or incorrect $JS.FC... routing can cause the sourcing relationship to behave badly. That is why a deployment can run smoothly for some time and then suddenly show high CPU, high memory, or a rapidly rising core NATS message rate after a burst.

Are account limits a fix?

Do not treat account limits as the primary fix for this class of problem. Limits may be useful for broader isolation goals, but they do not make an incorrect JetStream sourcing route correct.

If flow-control subjects, source-consumer API subjects, or stream data subjects are ambiguous or missing, correct the import/export configuration first.

Practical guidance

When cross-account JetStream sourcing causes CPU or message-rate spikes:

1. Compare JetStream stream metrics with core NATS subject activity.
2. Use subject reporting to find which control or data subject is multiplying.
3. Inspect the source account for repeated consumer-create activity.
4. Confirm that $JS.API.CONSUMER.CREATE.<stream>.> is exported and imported intentionally.
5. Confirm that $JS.FC.<stream>.> is exported and imported intentionally.
6. Keep source stream names unambiguous across the sourcing topology.
7. Avoid relying on broad mappings that could create loops or route flow-control traffic to the wrong place.

The most common practical fix is to correct the flow-control import/export mapping and remove any ambiguity in stream names or JetStream API prefixes.

Want help from the NATS experts? Meet with our architects to get help tailored to your use case and environment.