Can JetStream Work in a Fully Dynamic Edge Cluster?

A community member asked whether it makes sense to migrate from Core NATS to JetStream when an edge deployment can have anywhere from one to dozens of unstable nodes, with no central node that can be assumed to stay online.

The short answer: do not model that environment as one fully dynamic JetStream cluster unless you can provide stable JetStream membership and quorum. JetStream can still be useful, but the topology usually needs to separate routing from persistence.

Why a fully dynamic JetStream cluster is a poor fit

JetStream clustering uses RAFT for metadata and replicated stream state. RAFT is designed around a known group of peers that can form quorum. That has important consequences for highly dynamic edge environments:

JetStream peers do not automatically disappear from a RAFT group just because they are offline.
Scaling a JetStream cluster up or down is an administrative operation, not a mechanism for arbitrary membership churn.
If the cluster frequently shrinks below quorum, JetStream operations that require quorum can stop making progress.
A single logical cluster made entirely of unreliable members has no stable foundation for replicated persistence.

This does not mean JetStream is unsuitable for edge computing in general. It means that one JetStream cluster spanning all unstable edge members is usually the wrong abstraction.

If you have a stable hub, regional site, or subset of reliable nodes, JetStream clustering may fit there. Less stable nodes can connect through leaf nodes, and data can be moved with JetStream sourcing or mirroring where appropriate. See the NATS documentation for:

But if there is no stable core at all, the design needs different tradeoffs.

A useful pattern: separate Core routing from local JetStream storage

One pattern that may work is to run two NATS server processes per physical or logical edge member:

Gateway process: Core NATS only, no JetStream.
Storage process: standalone JetStream, connected to the gateway as a leaf node.

Conceptually:

1
edge member A
2
  gateway NATS process  <---- Core routes ---->  gateway NATS process
3
  local JetStream process -- leaf connection --> gateway NATS process
4

5
edge member B
6
  gateway NATS process  <---- Core routes ---->  gateway NATS process
7
  local JetStream process -- leaf connection --> gateway NATS process

In this model:

The gateway processes handle communication and routing.
Each storage process has its own standalone JetStream domain. Give every member a unique domain name so it can be addressed individually.
The storage processes do not form one JetStream cluster with each other.
Each local JetStream server can create streams and consumers with a replica count of 1 (replicas: 1), which is the only option on a single-server JetStream domain.
You avoid coupling JetStream RAFT quorum to the full, unstable edge membership.

This provides local persistence and JetStream client semantics at each member, but it is not the same as a replicated, highly available JetStream cluster. If the local storage process or disk is lost, a replicas=1 stream does not have another JetStream replica to recover from.

Example configuration sketch

The following is an illustrative shape, not a complete production configuration. Authentication, authorization, TLS, monitoring, storage paths, and operational controls need to be designed for your environment.

A Core gateway process might look like this:

1
listen: 192.0.2.10:4222
2
server_name: gateway-a
3

4
accounts: {
5
  $SYS: {
6
    users: [{ user: sys, password: pass }]
7
  }
8
  EDGE: {
9
    # No jetstream directive here: the gateway is a Core-only routing process.
10
    users: [
11
      { user: app-user, password: app-pass },
12
      { user: leaf-user, password: leaf-pass }
13
    ]
14
  }
15
}
16

17
cluster {
18
  name: edge-core
19
  listen: 0.0.0.0:4248
20

21
  routes: [
22
    nats://192.0.2.10:4248,
23
    nats://192.0.2.11:4248
24
  ]
25
}
26

27
leafnodes {
28
  port: 7422
29
}

A local standalone JetStream process on the same member might look like this:

1
listen: 127.0.0.1:4111
2
server_name: storage-a
3

4
accounts: {
5
  $SYS: {
6
    users: [{ user: sys, password: pass }]
7
  }
8
  EDGE: {
9
    jetstream: enable
10
    users: [{ user: app-user, password: app-pass }]
11
  }
12
}
13

14
# The jetstream block enables JetStream and sets a unique domain for this member.
15
jetstream {
16
  store_dir: /var/lib/nats/jetstream
17
  domain: storage-a
18
}
19

20
leafnodes {
21
  remotes: [
22
    {
23
      urls: ["nats://leaf-user:leaf-pass@192.0.2.10:7422"]
24
      account: EDGE
25
    }
26
  ]
27
}

The important detail is the account field on the leaf node remote. The user in the URL determines which account the remote side authenticates into. The account field determines the local account used by the leaf connection.

Leaf node connections connect accounts one-to-one. The account names on each side do not have to match, but being explicit avoids surprising behavior.

Common leaf node account mistake

If messages appear on the Core side but do not flow to the leaf side, check the account binding first.

For example, this is incomplete if you expect the local EDGE account to participate:

1
leafnodes {
2
  remotes: [
3
    { urls: ["nats://leaf-user:leaf-pass@192.0.2.10:7422"] }
4
  ]
5
}

Prefer this:

1
leafnodes {
2
  remotes: [
3
    {
4
      urls: ["nats://leaf-user:leaf-pass@192.0.2.10:7422"]
5
      account: EDGE
6
    }
7
  ]
8
}

Also remember:

Do not use the $SYS account for application traffic.
Accounts are isolated from each other.
Client listen ports, route ports, and leaf node ports should be distinct.
nats server report leafnodes and nats trace can help verify connectivity and subject interest.

How clients access local or remote JetStream domains

With standalone JetStream per member, each storage process has its own JetStream domain, and each domain name must be unique so a specific member can be addressed unambiguously. By default, a client’s JetStream context uses the JetStream available through the server it connects to: if it connects directly to a storage process, that is the local domain; if it connects to a Core gateway process, there is no local JetStream, so the client must name a domain explicitly.

A client can target a specific JetStream domain when it needs to interact with a remote member’s streams, assuming the topology and account permissions allow that access.

That means an offline member is not automatically limited to only what it saw while online. Depending on your design, a process can later read from another member’s JetStream domain. However, that is different from saying that all streams are automatically replicated everywhere.

Replication is not automatic in this pattern

The local-standalone pattern avoids a fragile global JetStream RAFT group, but it also means JetStream will not automatically replicate every member’s streams to every other member.

You have a few options, each with tradeoffs:

Use JetStream sourcing where membership is known

JetStream sourcing can copy messages from one stream into another. This can be useful when relationships between streams are known and managed.

However, sourcing is not a fully dynamic membership system. If nodes come and go arbitrarily, an administrative controller or operator process may need to add, remove, and reconcile source configuration.

Build a replication agent

Another option is an application-level replication agent. Such an agent can:

Discover reachable members.
Read from remote streams.
Track stream sequence numbers.
Resume from the last copied sequence after reconnecting.
Copy messages into local streams according to your conflict, deduplication, and retention rules.

This gives you control, but it moves part of the distributed data problem into your software.

Use overlapping ingress subjects intentionally

Be careful with stream subjects. If the same published subject is captured by streams in multiple domains, messages may be ingested in multiple places. That may be exactly what you want for status fanout, or it may create duplicates and reconciliation work.

For many edge systems, it is safer to design clear subject ownership, for example by including node identity or domain identity in the subject hierarchy.

When to choose Core NATS only

If the environment cannot provide any stable JetStream placement, and if local persistence does not add enough value, Core NATS plus application-level delivery mechanisms may still be the more honest design.

Core NATS is lightweight and flexible, but it does not provide JetStream persistence, durable consumers, or replay by itself. If you need stronger delivery confidence on top of Core NATS, you will need to design those semantics explicitly in the application.

Also consider the cost of a full mesh. Even without JetStream RAFT, a topology where every server connects to every other server can become operationally difficult as node count and network instability increase. If you can introduce a hub-and-spoke model with stable hubs, the overall NATS design usually becomes simpler.

Practical guidance

For highly unstable edge clusters with no stable core:

Avoid one JetStream cluster across all edge members.
Use Core NATS for dynamic communication if that fits your network model.
Consider a local standalone JetStream process per member when local persistence and replay are useful.
Connect local JetStream processes to gateway processes with leaf nodes.
Set a unique JetStream domain on each member so members can be addressed individually.
Be explicit about leaf node account mapping.
Treat cross-node stream replication as a separate design problem.

JetStream can still be valuable in this kind of architecture, but the value comes from carefully scoped persistence domains rather than pretending that an unstable full mesh can behave like a stable replicated storage cluster.

Want help from the NATS experts? Meet with our architects to get help tailored to your use case and environment.