Designing Tenant-Isolated Edge Cold Starts with JetStream

A community member asked how to cold start single-tenant edge environments from a multi-tenant NATS and JetStream deployment, while ensuring each edge instance only receives data for its tenant.

This is a common design problem: JetStream can distribute live data very well, but a cold start often needs historical state that is no longer retained in the live stream. The safest design is to separate those two concerns.

Short answer

Use separate mechanisms for live replication and cold-start backfill:

Use filtered JetStream sources for live tenant data; a mirror works only when the edge stream is a strict one-to-one copy of a single cloud stream.
Do not base source or mirror workflows on Interest retention streams.
Decide between a single shared multi-tenant cloud stream and one stream per tenant based on tenant count; both are valid, with different tradeoffs.
Use JetStream domains when an edge environment and the cloud both have JetStream enabled.
Use a custom backfill service or a dedicated backfill stream for state that is no longer in the live stream.
Prefer chunked messages in regular streams over Object Store when the data needs to participate in stream sourcing or mirroring workflows.
Use tenant accounts and subject-level permissions for isolation; JetStream domains are for addressing JetStream deployments, not for tenant security.

Separate live data from cold-start state

A live event stream and a complete tenant snapshot have different lifecycle requirements.

A live stream usually has bounded retention. It may retain only the recent window of data needed by online services. A cold-starting edge instance, however, may need the full current state for its tenant, including data older than the live stream retention window.

Trying to make one stream satisfy both requirements usually leads to tradeoffs that are hard to operate:

If retention is too short, cold starts miss data.
If retention is effectively forever, storage and replay time can grow without bound.
If Interest retention is used to clean up automatically, it is not a good foundation for source or mirror based replication.

A more robust pattern is:

Keep a bounded live stream for current tenant events.
Keep authoritative state elsewhere, such as a database or compacted application state.
Provide an explicit cold-start backfill path for reconstructing edge state.
Use deduplication or idempotent application logic to handle overlap between backfill and live data.

Use filtered sources or mirrors for live edge replication

JetStream sources and mirrors can replicate stream data from one JetStream deployment to another. They can also use subject filters, which is useful when subjects include a tenant identifier such as:

1
my.data.{tenant_id}.>

For a tenant uuid1, an edge stream can source only:

1
my.data.uuid1.>

This is a natural fit for distributing live data from a cloud stream to a tenant edge environment.

Source and mirror are not interchangeable. A mirror copies exactly one origin stream; a mirrored stream cannot be published to directly and cannot also pull from other streams. A stream configured with sources can aggregate multiple origin streams and can also accept its own direct publishes. Because the cold-start design below combines live tenant data with separately produced backfill data, an edge stream that aggregates both should use sources. Reserve mirrors for the case where the edge stream is a strict one-to-one copy of a single cloud stream.

However, sources and mirrors only replicate data available in the origin stream. They do not reconstruct messages that were already removed by retention. They also remember progress; they are not a magic snapshot mechanism for data that no longer exists in JetStream.

For streams that will be sourced or mirrored, use a retention policy that fits that workflow, typically Limits retention. Do not rely on Interest retention as the basis for source or mirror replication.

See the NATS documentation on JetStream sources and mirrors for the current configuration model.

Choosing the cloud stream layout

There are two valid ways to lay out the cloud side, and the right choice depends mostly on how many tenants you expect:

One shared multi-tenant stream capturing my.data.>. Each edge sources only its tenant’s filtered subjects. This keeps the cloud stream count low and constant, but every tenant shares one stream’s storage, replication, and limits, and isolation depends entirely on subject filtering plus account permissions.
One stream per tenant. Each tenant gets an independent stream, which allows per-tenant retention, limits, and placement and makes isolation easier to reason about. The cost is that the number of streams grows linearly with the number of tenants.

If the tenant population is large — on the order of tens of thousands or more — neither option is free. Per-tenant streams and per-tenant accounts can reach practical operational limits on a single cluster, and a single shared stream concentrates all tenant load in one place. Benchmark the chosen layout against realistic tenant counts, message rates, and storage, and be prepared to shard tenants across multiple clusters or accounts if one cluster cannot hold them all.

Use JetStream domains for cloud and edge JetStream

If the cloud and the edge both run JetStream, configure distinct JetStream domains. Domains let clients and servers address the intended JetStream deployment when more than one JetStream environment is reachable, such as across leaf node connections.

A common shape is:

Cloud NATS cluster with JetStream domain for hub data.
Edge NATS server or cluster with its own JetStream domain.
Edge stream that sources filtered tenant data from a cloud stream.

Domains are not tenant isolation. They are an addressing and routing tool for JetStream APIs. Tenant isolation should come from accounts, imports and exports, and subject permissions.

Cold start needs an explicit backfill path

If the live stream does not contain the full tenant history, the edge needs another source of truth for cold start. NATS can still help move that data, but application logic must decide what to export, how to chunk it, and how to handle duplicates.

A practical cold-start sequence can look like this:

Start the edge NATS connection and leaf node path.
Create or resume the edge stream that sources live tenant data from the cloud, but do not apply live data to the local database until the backfill boundary is understood.
Trigger a tenant backfill job.
Have the backfill job publish chunks into a dedicated tenant backfill stream, or onto a backfill subject the edge consumes directly if that is simpler.
Load the chunks into the local embedded database.
Start consuming live data from the edge stream.
Use idempotent writes, CRDT semantics, or Nats-Msg-Id based deduplication to tolerate overlap.

If ordering is not important because the application uses CRDTs or another eventually consistent merge model, this becomes easier. You still need a clear backfill boundary and duplicate handling, but you do not need NATS to provide a total order across database export data and live stream data.

Request-reply can trigger work, but it is not the transfer protocol

Core NATS request-reply is reasonable for triggering a backfill job, but avoid using the reply payload to return a large export. Large data should be streamed as messages or chunks.

If multiple services can perform the same export, use a queue subscription for the request subject so that one member of the queue group receives the request. Without a queue group, every matching subscriber can receive the request and may start redundant work.

That said, queue delivery is not the same as end-to-end exactly-once export semantics. A worker can accept a request and fail before completing the export. A client can retry and create another job. For heavy exports, make the operation idempotent:

Include a stable export job ID.
Store export status by tenant and job ID.
Make retries resume or return the existing job.
Use deterministic message IDs where possible, such as Nats-Msg-Id, so duplicate chunks can be discarded.

For latency-sensitive cases, do not assume a queue group will always choose the nearest responder. Make locality explicit. For example, first ask a local edge service on a local-only subject or account. If no response arrives within a short timeout, fall back to a cloud service. This makes the preference clear and avoids accidental duplicate exports.

Prefer regular streams for backfill chunks

Object Store can be useful for storing large objects, but for this design a regular stream of chunked messages is often easier to combine with JetStream sourcing, filtering, deduplication, and progress tracking.

A backfill stream could contain messages such as:

1
my.backfill.uuid1.{job_id}.{chunk_number}

Each chunk can include headers for:

tenant ID
export job ID
chunk number
total chunks, if known
source snapshot version or timestamp
Nats-Msg-Id for deduplication

The edge can consume this stream, apply chunks to the local database, and then switch to the live stream. Stream information can also be used to inspect replication progress when sourcing is involved.

This does not remove the need for application-level consistency rules. It gives you NATS-native building blocks for moving the backfill data in a way that is observable and retryable.

Tenant authorization: isolate with accounts and permissions

Authentication method and authorization policy are separate concerns. Whether clients authenticate with JWTs, NKEYs, or another supported mechanism, tenant isolation comes from accounts and permissions.

A common model is one account per tenant. Accounts are the strongest isolation boundary in NATS: each account has its own subject space, and a client in one account cannot see or publish another account’s subjects unless an explicit export and import grants it. With account-per-tenant, cross-tenant isolation is structural — the account for tenant uuid1 has no path to my.data.uuid2.> at all. Within the tenant account, use strict publish and subscribe permissions to scope what the tenant may do, for example allowing only:

1
my.data.uuid1.>
2
my.backfill.uuid1.>

If instead all tenants share a single account and isolation relies only on subject permissions, you must explicitly allow each tenant’s own subjects and deny the others, such as:

1
my.data.uuid2.>

That single-account model is more error-prone and is generally not recommended for hard multi-tenant isolation; account-per-tenant is the safer default.

Also restrict JetStream API access, not only application data subjects. Source and mirror creation involves JetStream API calls, and filter subjects are part of those requests. The NATS API reference describes the subjects used for stream source and mirror operations: NATS API Reference.

The important rule is that the tenant account should only be able to create or access filtered consumers and streams for that tenant.

Local-only edge data must be protected by permissions

A leaf node connects NATS environments. It is not, by itself, a data-loss-prevention boundary.

If customers can publish local data that must never go to the cloud, put that data on local-only subjects or streams and enforce that policy with account imports, exports, and publish permissions. Do not export those subjects to the cloud account, and do not grant permissions that allow the local client to publish them into cloud-routed subjects.

If local-only data and cloud-replicated data share similar subject patterns, be especially careful. Clear subject taxonomy is part of the security model.

Multiple edge containers and leaf nodes

If an edge environment needs high availability for NATS itself, run an actual NATS server cluster at the edge and connect it to the cloud using leaf node connections. A NATS cluster manages its own leadership through the clustering protocol, including JetStream stream and consumer leadership through Raft, so you do not need to implement leader election for NATS itself.

If the edge containers are only application clients with ephemeral local databases, then leader election or single-writer behavior is an application design question. NATS can help with queue groups and idempotent message processing, but it cannot decide which application instance has the authoritative local state unless the application defines that rule.

For backfill services, prefer designs that are safe when triggered more than once:

queue requests to reduce duplicate work
use job IDs
make exports resumable
deduplicate chunks
make database writes idempotent

Recommended design checklist

For a tenant-isolated edge cold start design, validate these choices in a proof of concept:

Live stream uses retention compatible with sourcing or mirroring.
Cloud stream layout (one shared multi-tenant stream or one stream per tenant) is chosen and benchmarked for the expected tenant count.
Edge stream uses a subject filter for exactly one tenant.
Cloud and edge JetStream deployments use distinct domains.
Cold-start data is separate from live stream data.
Backfill is chunked through a regular stream or another explicit NATS workflow.
Request-reply is only used to trigger or coordinate the export, not to carry the full payload.
Duplicate exports and duplicate chunks are safe.
Tenant authorization restricts both data subjects and JetStream API subjects.
Local-only edge subjects are not exported to the cloud.

Conclusion

JetStream sources, mirrors, leaf nodes, and domains are strong building blocks for live tenant data distribution. They do not replace an explicit cold-start strategy when the full tenant state is no longer retained in JetStream.

The practical approach is to separate live replication from historical backfill, use subject-filtered streams so each edge replicates only its own tenant’s data, chunk backfill data through regular streams when possible, and make the export process idempotent. Tenant isolation itself should be enforced with per-tenant accounts and subject permissions, while domains should be used only to distinguish cloud and edge JetStream deployments.

Want help from the NATS experts? Meet with our architects to get help tailored to your use case and environment.