Partitioned Consumer Groups in NATS JetStream: Ordered, Scalable Processing

TL;DR: Partitioned consumer groups give NATS JetStream the Kafka-style guarantee of per-key ordering that survives horizontal scaling — every message for a given key is pinned to one worker, while different keys run in parallel. The Orbit pcgroups library packages the pattern into a consumer-group API with static and elastic flavors.

Watch the video for a quick walkthrough, or keep reading for the full breakdown.

The Problem: Ordering and Scale Pull in Opposite Directions

Most stream-processing work only needs ordering among related messages — all the events for one customer, one account, one device — not across the whole stream. Two events for customer alice, orders.alice.buy then orders.alice.pay, must be processed in that sequence. Two events for two different customers have no such constraint and can safely run in parallel.

The trouble is that the usual scaling tools throw ordering away. An ordinary JetStream consumer (a view onto a stream that tracks what has been read and acknowledged) load-balances messages across whatever workers are pulling. That is exactly what you want for independent messages — and exactly what breaks alice. Put two workers on the stream and pay can land on the idle one and finish before buy has even been picked up. You charged the card before the cart was confirmed.

The obvious fixes each fail in their own way. One worker still processes concurrently if it pulls more than one message at a time. Pinning it to max_ack_pending = 1 — hold exactly one un-acknowledged message at a time — restores strict order, but serializes the entire stream: bob waits behind alice for no reason, and a lone worker is a single point of failure. You cannot even add a hot standby, because the moment a second worker takes real load you are load-balancing again and back to the reordering bug. You traded away throughput and availability to get ordering.

Teams coming from Apache Kafka know the shape of the real answer: consumer groups with partitions. Split the stream into buckets by key, assign each bucket to exactly one worker, and you get ordering within a bucket while running many buckets in parallel. NATS builds that same model out of JetStream primitives — no external coordinator.

Partitioned Consumer Groups: The Solution

A partitioned consumer group routes every message that shares a partition key (the value used to decide a message’s bucket, e.g. a customer id) to the same partition, and pins each partition to a single worker. It rests on two server features — deterministic subject partitioning (since NATS Server 2.10) and pull consumer priority groups with the pinned_client policy (since 2.11) — and is delivered as a ready-made library, pcgroups, in Synadia’s Orbit collection. pcgroups ships a demo CLI called cg and a Go API, in two flavors: static and elastic. The whole pattern requires NATS Server 2.11+.

How It Works

The Vocabulary: Stream, Group, Member, Partition

Four nested ideas, and mixing them up is where the confusion lives:

• Stream — the data. Named for what it holds (ORDERS, everything under orders.*). This is what gets partitioned.
• Group — one kind of work over that stream, named for the job (fulfillment, billing, analytics). One stream can feed many groups, each reading it independently and tracking its own progress. A group is never named for the stream’s data.
• Partition — a filtered slice of the subject you can pin ordering on, e.g. 0.orders.alice.buy. Every key hashes to exactly one partition.
• Member — one named consumer plus the worker draining it, owning a fixed set of partitions. All members in a group run at once, each on a different set of partitions.

Deterministic Subject Partitioning

Partitioning starts with a subject transform (a server rule that rewrites a message’s subject as it flows through). The {{partition(N, token...)}} function hashes the chosen subject tokens — the dot-separated segments of a subject like the alice in orders.alice.buy — into a number from 0 to N-1, and that number is prepended as a new leading token.

Under the hood the server uses an FNV-1a 32-bit hash modulo the partition count. Because the hash is deterministic, identical token values always land in the same partition — the property that keeps every message for a given key together. So orders.alice.buy might become 2.orders.alice.buy, and alice will hash to 2 forever. That leading token is what every downstream consumer filters and pins on.

Pinned Clients: One Active Worker per Partition, With Failover

Each partition needs exactly one active worker. pcgroups gets this from a priority group on a pull consumer using the pinned_client policy (the server calls the group PCG). The server designates one pinned client to receive a group’s messages while other bound clients wait on standby. Delivered messages carry a Nats-Pin-Id header the client must echo on every pull; a pull whose id does not match the current pin is rejected with a 423 status, and that client rejoins the standby pool. If the pinned client goes quiet past its PinnedTTL, or is unpinned, the server re-pins to a standby — failover (shifting work to a backup when the active worker stops) with no external leader-election.

Here is the nuance most people get wrong: “one active, many standby” applies to redundant instances of a single member, not to members relative to each other. All members run at once, each draining its own partitions. Run two processes both joining as member m1 and the server pins one, leaving the other a hot standby for m1 specifically. Members shard the work; standby instances back a single member up — two different axes.

Members vs. Partitions

Partition count and member count are separate dials, and effective parallelism is min(partitions, members). The partition count is fixed at creation and is the immutable ceiling on how many members can do useful work at once; members are cheap to add or remove up to that ceiling. With 4 partitions and 2 members, each member simply owns two partitions:

1
2 members, 4 partitions:   m0[p0,p1]  m1[p2,p3]           ← half capacity
2
add a member  →            m0[p0]     m1[p2,p3]  m2[p1]   ← one partition moves
3
add another   →            m0[p0]     m1[p2]  m2[p1]  m3[p3]  ← 1:1, maxed out

The assignment is deliberately stable (contiguous blocks, leftovers round-robin), so growing the group re-homes as few partitions as possible. A 5th member owns zero partitions and idles as a standby — where scale-out stops and redundancy begins.

One subtlety worth internalizing: a member drains all its partitions serially. pcgroups puts every partition a member owns into a single FilterSubjects, creates one consumer, and calls Consume once — and the Go client dispatches that callback one message at a time. Piling more partitions onto a member is parked ownership, not added throughput. The only way to turn parked partitions into real parallelism is to hand one to another member.

Static vs. Elastic

Both flavors give the identical guarantee — each key pinned to one serial worker, ordered within a key, parallel across keys — and both store their config (member list, partition count) in a JetStream KV bucket. They differ in where the consumers live and whether membership can change at runtime.

Static attaches consumers directly to your stream, so the stream must already be partition-stamped by a subject transform. Each member gets one durable consumer named <group>-<member>, filtered to its partitions. Membership is frozen: to change it you delete and recreate the group. Lower latency, fewer server resources, any ack policy.

Elastic works on any stream — no pre-existing partition token needed. The library sources your origin stream into a hidden work-queue stream named <stream>-<group>, applying the {{partition(...)}} transform as it copies each message in, so your producers stay untouched. Members consume from that work-queue stream, and because the library owns it, it reassigns partition filters live. AddMembers/DeleteMembers just edit the KV config; every running member watches the key and rebalances itself. Explicit ack is required.

The one thing neither flavor will change live is the partition count — both reject it and tell you to delete and recreate the group. Elastic gives you elastic members over a fixed number of partitions.

Code Examples

Bash (cg CLI) — A static group over a partition-stamped stream

# Partition the stream up front: prepend a deterministic token 0..9 hashed from the customer id.
nats stream add ORDERS --subjects="orders.*" \
  --transform-source="orders.*" \
  --transform-destination="{{partition(10,1)}}.orders.{{wildcard(1)}}" --defaults

# Create a static group over 10 partitions with two members, balanced.
# Positional args: <stream> <name> <max-members> <members...>; the filter is a flag.
cg static create balanced ORDERS fulfillment 10 m1 m2 --filter '>'

# Start a worker per member (run each in its own process; add more instances of a
# name for hot standbys). Only one instance of a member is pinned at a time.
cg static consume ORDERS fulfillment m1 --sleep 25ms
cg static consume ORDERS fulfillment m2 --sleep 25ms

# Inspect configured vs. currently active members, and step a pinned instance down.
cg static info ORDERS fulfillment
cg static step-down ORDERS fulfillment m1

max-members (10) is the partition count — it must match the partition(10,...) in the transform. m1 and m2 split the 10 partitions between them, leaving headroom to add up to eight more members later without re-partitioning.

Bash (cg CLI) — An elastic group on a plain, unpartitioned stream

# A plain stream — no partition token in the subject.
nats stream add ORDERS --subjects="orders.*" --defaults

# Elastic group over 10 partitions; it builds the hidden ORDERS-fulfillment
# work-queue stream that stamps the partition token as it sources.
cg elastic create ORDERS fulfillment 10

# Membership is live: add members any time, up to the max. A member becomes active
# only once both (a) it is in the config and (b) an instance is running under its name.
cg elastic add ORDERS fulfillment m1 m2
cg elastic consume ORDERS fulfillment m1
cg elastic consume ORDERS fulfillment m2

# Drop and re-add live — survivors pick up the orphaned partitions, no teardown.
cg elastic drop ORDERS fulfillment m1

The origin stream and its producers never change; partitioning happens in the sourced ORDERS-fulfillment work-queue stream the library manages.

Go — A static group with the pcgroups API

1
import "github.com/synadia-io/orbit.go/pcgroups"
2

3
// Static: assumes ORDERS is already partition-stamped by the subject transform.
4
// maxMembers (10) == the partition count baked into {{partition(10,...)}}.
5
_, err := pcgroups.CreateStatic(ctx, js, "ORDERS", "fulfillment",
6
    10, []string{">"}, []string{"m1", "m2"}, nil)
7

8
// Join as a member. The handler MUST be idempotent: pinning is affinity + failover,
9
// not a lock, so a failover window can hand the same message to two workers.
10
// The last arg is a ConsumerConfig template (values like filters and priority
11
// groups are overwritten by the library); pass a zero value to accept defaults.
12
_, err = pcgroups.StaticConsume(ctx, js, "ORDERS", "fulfillment", "m1",
13
    func(msg jetstream.Msg) { /* process */ _ = msg.Ack() }, jetstream.ConsumerConfig{})

The library strips the partition token before your handler runs, so existing callback code works unchanged.

Go — An elastic group with the pcgroups API

1
// Elastic works on a plain ORDERS stream. Instead of a pre-stamped subject you
2
// pass PartitioningFilters: filter "orders.*" and hash wildcard index 1 (the
3
// customer) into 10 partitions. The two 0s are the buffer caps (0 = unbounded).
4
_, err := pcgroups.CreateElastic(ctx, js, "ORDERS", "fulfillment", 10,
5
    []pcgroups.PartitioningFilter{{Filter: "orders.*", PartitioningWildcards: []int{1}}}, 0, 0)
6

7
// Membership is live — add members any time, up to maxMembers.
8
_, err = pcgroups.AddMembers(ctx, js, "ORDERS", "fulfillment", []string{"m1", "m2"})
9

10
// Consuming is the same call shape as static, but the config template's AckPolicy
11
// MUST be explicit — elastic rejects anything else.
12
_, err = pcgroups.ElasticConsume(ctx, js, "ORDERS", "fulfillment", "m1",
13
    func(msg jetstream.Msg) { /* process */ _ = msg.Ack() },
14
    jetstream.ConsumerConfig{AckPolicy: jetstream.AckExplicitPolicy})

The member and handler shape is identical to static; only the setup differs.

When to Use This

Ordered per-key processing at scale — Partition by customer, account, or device id so all of one key’s events are processed in order by a single worker, while different keys run in parallel. The canonical use: you need both ordering and throughput, and partitioning is how you stop having to choose.

Single-active-worker with hot standby — Run redundant instances of a member and the pinned-client mechanism keeps exactly one active, promoting a standby if it stalls past PinnedTTL. This replaces a whole class of leader-election machinery for jobs that must run on exactly one node at a time.

Isolating a hot partition (a “whale”) — When one key fires ten times the volume of the rest, a balanced split can strand a quiet key behind the flood on the same member. Hand-assign partitions with a custom mapping (member:partition,..., every partition appearing once) so the whale gets a member to itself. Static bakes the mapping in at creation; elastic can re-set it live.

Membership that changes over time — Reach for elastic when workers come and go, when you cannot add a transform to the source stream, or when several groups share one stream — at the cost of a buffered second copy.

Tips and Gotchas

• Pinning is affinity plus failover, not a distributed lock. There are windows where a client still believes it is pinned after the server has re-pinned, and messages already in a client’s buffer cannot be revoked. Make handlers idempotent (safe to run more than once with the same result).
• The partition count is frozen forever — in both flavors. Elastic makes membership live, not the partition count. Size partitions for peak future parallelism, because raising it later means deleting the group and re-hashing every key.
• A member does not parallelize its own partitions. They share one serial worker — add a member to get real parallelism.
• Elastic requires explicit ack. The ack policy when consuming from an elastic group must be explicit; static has no such requirement.
• Cap the elastic work-queue above one second of throughput. When the buffer hits its max_buffered_msgs/max_buffered_bytes limit, sourcing pauses for at least a second to let consumers drain — set the cap too low and you will see periodic stalls.
• {{partition(...)}} cannot cross accounts. It is rejected in inter-account import transforms, where only {{wildcard(x)}} is allowed.

Wrapping Up

Partitioned consumer groups give NATS the guarantee that historically forced teams onto heavier infrastructure: per-key ordering that survives horizontal scaling, with automatic failover and no external coordinator. The recipe is deterministic subject partitioning to separate messages by key, plus pinned-client pull consumers to give each partition one active worker — and the Orbit pcgroups library wraps both into a static or elastic consumer-group API.

Two things to remember and the rest follows: pinning is affinity and failover, not exclusivity — so keep handlers idempotent — and the partition count is the one dial you size ahead of time, because it is the only one you cannot change without recreating the group. For more, see the Orbit partitioned consumer groups post and the NATS subject mapping docs.

Requires NATS Server 2.11+ (subject partitioning since 2.10). Uses the Orbit pcgroups library. See the official documentation for the full reference.