How to Map JetStream Subjects with External Metadata

A common community question is how to republish JetStream messages onto new subjects when the new subject depends on metadata that is not present in the original subject or payload.

For example:

• Devices publish events to incoming.{device-id}.{event-type}.
• Each device-id belongs to a group-id stored in an external database such as Redis.
• Downstream consumers should read from outgoing.{group-id}.{device-id}.{event-type}.
• Payloads should not change.
• Ordering must be preserved per device-id.
• Latency matters.

There are a few viable designs, but the right choice depends on whether the mapping can be expressed entirely in NATS configuration or requires an external lookup.

Short answer

If the subject transformation can be described statically, or by deterministic rules based only on subject tokens, use NATS subject mapping or JetStream subject transform capabilities where appropriate.

If the transformation requires a dynamic device-id to group-id lookup from an external database, use a custom mapper service. To scale that mapper while preserving per-device order, partition by device-id so all messages for the same device are handled by the same worker.

For very latency-sensitive paths, consider placing the mapper before the durable INCOMING stream and publishing directly to the OUTGOING stream. That avoids an extra JetStream hop, but it changes the durability and retry tradeoffs.

Why configuration-only subject mapping may not be enough

NATS supports subject mapping, including deterministic subject token partitioning. These features are a good fit when the new subject can be derived from the original subject using configured rules.

For example, configuration-based mapping can help when you need to route messages into partitions based on a subject token such as device-id.

It is usually not a good fit when the mapping requires a large, frequently changing lookup table such as:

1
device-123 -> group-a
2
device-456 -> group-b
3
device-789 -> group-a

If that table lives in an external database and changes over time, pushing every mapping into NATS configuration is likely to be operationally awkward. In that case, keep NATS responsible for routing and persistence, and put the database-dependent logic in an application-level mapper.

Option 1: Consume from INCOMING, transform, publish to OUTGOING

The simplest design is:

1. Create an INCOMING stream on incoming.>.
2. Run a mapper service that consumes from INCOMING.
3. For each message, parse device-id and event-type from the subject.
4. Look up group-id in the external database or local cache.
5. Publish the same payload to outgoing.{group-id}.{device-id}.{event-type}.
6. Store those messages in an OUTGOING stream on outgoing.>.

Conceptually:

1
incoming.{device-id}.{event-type}
2
        |
3
        v
4
mapper: lookup group-id for device-id
5
        |
6
        v
7
outgoing.{group-id}.{device-id}.{event-type}

This is straightforward and gives you a durable buffer before mapping. If the mapper is down, messages can remain in INCOMING until the mapper catches up.

The main limitation is scaling while preserving order.

A single mapper can preserve order, but may become a throughput bottleneck. A queue group of mapper instances can increase throughput, but messages for the same device-id may be processed by different workers. That can break per-device ordering once messages are republished to OUTGOING.

Option 2: Partition by device-id, then run one mapper per partition

A better scalable pattern is to partition the input by device-id before mapping.

NATS subject mapping supports deterministic subject token partitioning. The important property is that the same device-id is always mapped to the same partition.

Concretely, a mapping rule rewrites incoming.{device-id}.{event-type} to include a partition token derived by hashing device-id, for example incoming.{partition}.{device-id}.{event-type}. Because the hash is deterministic, every message for a given device lands in the same partition. The INCOMING stream still stores this partitioned subject (it matches incoming.>), and you create one durable consumer per partition, each filtering on a single partition token and consumed by exactly one worker at a time.

If you would rather not change the stored subjects, an alternative is to consume the whole INCOMING stream with a single consumer and partition in application code by consistent-hashing device-id into ordered worker lanes. Either way the requirement is the same: all messages for one device must travel a single ordered path.

Then run mapper workers so each partition is handled in order by a single worker at a time:

1
incoming.{device-id}.{event-type}
2
        |
3
        v
4
partition based on {device-id}
5
        |
6
        v
7
mapper worker for that partition
8
        |
9
        v
10
outgoing.{group-id}.{device-id}.{event-type}

This gives you parallelism across partitions while keeping all messages for a given device on the same ordered path.

The key implementation rule is: do not introduce concurrency inside a partition that can reorder messages for the same device-id. If a worker performs asynchronous publishes or concurrent database lookups, make sure completion and acknowledgement behavior cannot reorder republished messages for an individual device.

Option 3: Put the mapper before the durable stream

For the lowest-latency design, you may be able to skip the INCOMING stream entirely:

1. Devices publish to incoming.{device-id}.{event-type}.
2. NATS subject mapping partitions the incoming subject by device-id.
3. Core NATS mapper subscribers receive partitioned traffic.
4. Each mapper looks up or caches device-id -> group-id.
5. The mapper republishes the unchanged payload to outgoing.{group-id}.{device-id}.{event-type}.
6. The OUTGOING stream stores outgoing.> messages.

As in Option 2, ordering depends on each partition being served by a single active subscriber. If you run multiple mapper instances for availability, treat them as failover for a partition rather than load-balancing one partition across them. A Core NATS queue group spread over a single partition would distribute that partition’s messages across instances and reorder them.

This removes the durable INCOMING hop from the hot path. That can reduce latency and storage work, but it also means there is no durable pre-mapping buffer.

In this design the original device still wants a JetStream publish acknowledgement, but there is no INCOMING stream to produce one. The pattern that makes this work is to preserve the original message’s reply subject. When a client performs a JetStream publish, it sends the message with a reply inbox and waits for a publish acknowledgement (PubAck) on it. If the mapper receives that message over Core NATS with its reply subject intact and republishes it with a plain Core NATS publish to outgoing.{group-id}.{device-id}.{event-type} — keeping the reply subject and not changing the payload — then the OUTGOING stream sends its PubAck directly to the original device. The device’s publish call completes as if it had published straight to a stream.

This works only under specific conditions, so confirm them for your client libraries and configuration:

• The mapper republishes with a plain Core NATS publish and preserves the original reply subject. It should not consume-and-acknowledge a JetStream message and then publish a brand-new one, because that decouples the device’s acknowledgement from the OUTGOING write.
• The mapper does not introduce publish-expectation headers (such as an expected stream or expected last sequence) that the OUTGOING stream would reject, and does not alter a Nats-Msg-Id header if the publisher relies on it for deduplication.
• The publisher treats a missing acknowledgement as a timeout and retries. If the mapper is unavailable or drops the message before it reaches OUTGOING, no PubAck is ever produced, and there is no durable record of the message until OUTGOING stores it.

This pattern is attractive when:

• The mapper is lightweight and highly available.
• Device-to-group lookups are cached effectively.
• Publishers can retry when no acknowledgement is received.
• The system can tolerate not having a durable record in INCOMING before mapping.

It is less attractive when:

• You need a durable audit trail of raw incoming messages.
• The mapper or metadata store may be unavailable for extended periods.
• You want to replay the original incoming stream through a new mapping implementation.

Is this a fit for NATS Workloads?

NATS Workloads may be a way to run and operate the mapper component in environments where you are already using it, but the core design question is independent of Workloads.

The metadata-dependent transformation still needs application logic somewhere, because the group-id comes from an external lookup. The important architectural choices are:

• whether mapping happens before or after a durable INCOMING stream;
• how device-id partitioning is performed;
• how mapper workers preserve per-device order;
• how cache misses, database latency, retries, and failures are handled.

Handling group changes

If a device can move from one group to another, define the expected behavior explicitly.

Questions to answer include:

• Should messages already received continue to use the old group, or should they use the latest group at processing time?
• How quickly must a group change take effect?
• How are mapper caches invalidated or refreshed?
• Can a device’s messages appear under two group subjects during a transition?

These rules are application-specific. NATS can route and persist the resulting subjects, but the semantics of metadata changes belong in the mapper and metadata system.

Practical recommendation

For a dynamic device-id to group-id mapping, start with a custom mapper service rather than trying to encode the entire mapping table in NATS configuration.

If durability before mapping matters, consume from an INCOMING stream and publish to an OUTGOING stream. If latency is the higher priority and publishers can retry, consider mapping before the stream and publishing directly into OUTGOING.

In both designs, use deterministic partitioning by device-id when you need to scale horizontally without losing per-device ordering.

Want help from the NATS experts? Meet with our architects to get help tailored to your use case and environment.