Distributed Counters in NATS JetStream: A CRDT-Based Approach to Global Counting

Watch the video for a quick walkthrough, or keep reading for the full breakdown.

The Problem With Counting at Scale

Counting events sounds like one of the simplest things a distributed system ever has to do. Add one on a page view. Subtract one when a refund processes. Report the total.

Actually building that at global scale is where it gets painful. A relational counter protected by a lock gives you perfect ordering and an exact total — but every increment waits for coordination, and a partition between your regions can freeze writes entirely. Netflix found the problem interesting enough to build a dedicated Distributed Counter Abstraction: 75,000 increments per second, single-digit millisecond latency, no central lock in the critical path.

Every solution to this problem has the same shape: write locally, tolerate divergence, merge when the network heals. Teams implement it with regional Redis clusters, Kafka delta logs, or sharded aggregators — but they all end up coding the same merge logic. What they actually need is an increment that is order-independent by construction, safe to retry, and convergent across replicas without any custom reconciliation code.

The Counter Stream: A NATS-Native Solution

NATS Server 2.12 adds exactly that primitive. A JetStream stream configured with AllowMsgCounter: true accepts only messages carrying a Nats-Incr header — a signed integer delta — and rewrites the body of the most recent message on each subject with the running total. Under the hood it behaves as a CRDT (conflict-free replicated data type) — a data structure where multiple servers can update their own copies independently and still arrive at the same result when they merge. Addition and subtraction are already commutative operations, so a stream that only accepts signed deltas is automatically CRDT-safe. Order of arrival stops mattering. Retries stop corrupting state. Sources across regions converge to the same total.

How It Works

Enabling Counter Mode on a Stream

Turning a stream into a counter is a one-line config change. Set AllowMsgCounter to true when you create the stream, and set AllowDirect alongside it so the client can batch-read values:

1
stream, err := js.CreateOrUpdateStream(ctx, jetstream.StreamConfig{
2
    Name:            "COUNTERS",
3
    Subjects:        []string{"counters.>"},
4
    AllowMsgCounter: true,
5
    AllowDirect:     true,
6
})

Every subject that matches the stream is now a counter. There is no separate “counter object” — the stream is the counter registry.

Incrementing with the Nats-Incr Header

A publish to a counter subject must carry a Nats-Incr header whose value matches the regex ^[+-]\d+$. Valid examples: +1, -5, +1000000000000. The server reads the last message on that subject to get the current total, applies the delta with big.Int arithmetic, and stores a new message with {"val":"<new total>"} as the body. The PubAck that comes back includes the post-increment value in its val field, so you never need a separate read to know where the counter stands.

Arbitrary Precision by Default

The server uses big.Int internally. Counter totals can exceed 2^64 without overflow, and values are serialized as JSON strings rather than numbers — otherwise JavaScript clients, capped at 53-bit safe integers, would silently lose precision on large totals.

Multi-Region Aggregation Through Sources

This is where the CRDT property earns its keep. A source (a JetStream feature that pulls messages from one stream into another, optionally with subject rewrites) lets you build an aggregation tree: regional counters feed continental counters feed a global counter.

When a counter stream sources from another counter stream, the server does not replay the sourced message’s Nats-Incr verbatim. It calculates the delta between the last recorded value for that source and the incoming value, then rewrites Nats-Incr to reflect only that delta. A Nats-Counter-Sources header maintained by the server records the last seen value for every source, keyed by {stream name}#{original subject}. Entries are never removed, even if you later drop a source — re-adding it would otherwise miscount.

Rejected Headers and Incompatibilities

A counter stream has strict rules about what else can coexist with it:

And once you turn AllowMsgCounter on, it cannot be turned off.

Code Examples

CLI — Create a Counter Stream and Increment

# Create a counter-enabled stream
nats stream add COUNTER --subjects "counter.>" --allow-counter

# Increment
nats req counter.page.views "" -H "Nats-Incr:+1" --raw
# {"stream":"COUNTER", "domain":"", "seq":1, "val":"1"}

# Another increment — the PubAck returns the new total
nats req counter.page.views "" -H "Nats-Incr:+100" --raw
# {"stream":"COUNTER", "domain":"", "seq":2, "val":"101"}

# Read the last stored value
nats stream get COUNTER --last-for "counter.page.views"
# Body: {"val":"101"}

Every nats req returns the post-increment total directly — no follow-up read is needed unless you want to inspect the stream message itself.

Go — Typed Client via orbit.go

The typed Go API lives in the orbit.go/counters module, published separately from nats.go so it can evolve independently. NewCounterFromStream validates both AllowMsgCounter and AllowDirect before it will operate.

1
package main
2

3
import (
4
    "context"
5
    "fmt"
6
    "math/big"
7

8
    "github.com/nats-io/nats.go"
9
    "github.com/nats-io/nats.go/jetstream"
10
    "github.com/synadia-io/orbit.go/counters"
11
)
12

13
func main() {
14
    nc, _ := nats.Connect(nats.DefaultURL)
15
    defer nc.Close()
16

17
    js, _ := jetstream.New(nc)
18
    ctx := context.Background()
19

20
    stream, _ := js.CreateOrUpdateStream(ctx, jetstream.StreamConfig{
21
        Name:            "COUNTERS",
22
        Subjects:        []string{"events.>"},
23
        AllowMsgCounter: true,
24
        AllowDirect:     true,
25
    })
26

27
    counter, _ := counters.NewCounterFromStream(js, stream)
28

29
    // Convenience AddInt for native ints
30
    total, _ := counter.AddInt(ctx, "events.orders", 1)
31
    fmt.Printf("orders = %s\n", total)
32

33
    // Add for arbitrary-precision big.Int
34
    big1T := new(big.Int)
35
    big1T.SetString("1000000000000", 10)
36
    total, _ = counter.Add(ctx, "events.revenue", big1T)
37
    fmt.Printf("revenue = %s\n", total)
38

39
    // Load reads the current total
40
    current, _ := counter.Load(ctx, "events.orders")
41
    fmt.Printf("orders (loaded) = %s\n", current.Val)
42
}

The Counter interface provides Add, AddInt, Get (full entry with source tracking), Load (current total only), and GetMultiple (streaming iterator across many subjects).

When to Use This

View counts and impression tracking — every page hit bumps a subject like counter.page.<slug>. Eventual consistency is fine: readers never notice that the dashboard lagged the true total by a few hundred milliseconds during a partition, and the count is correct once the network heals.

Edge rate limiting — each region maintains its own token-bucket counter; totals converge through sources so quota enforcement is approximately global, but every increment stays local. You get low-latency rate checks without a central lock.

Per-region metrics aggregation — regional counters feed a continental counter, which feeds a global counter. The CRDT semantics handle the merge automatically, so your metrics pipeline looks like three JetStream streams connected by Source configs instead of a custom aggregation service.

Billing meters and leaderboards — anything that must survive partitions and reconnect cleanly. The combination of at-least-once delivery on JetStream and idempotent CRDT merges means retries never double-charge a customer, and a failed replica rejoining the cluster picks up where it left off.

Tips and Gotchas

• One-way switch — AllowMsgCounter cannot be disabled once enabled. Plan the rollout carefully.
• No mirrors — counter streams cannot be mirrors. If you need aggregation across regions, use sources, which do delta-rewriting correctly. Mirrors would replay raw Nats-Incr headers and double-count.
• Reset takes thought — subject purge does not replicate across sourced streams. To reset a replicated counter, publish a negative delta equal to the current total at the tier you want to reset, then purge that message. Never use Nats-Rollup — it would destroy the entire counter across the topology.
• Retention policy is locked to Limits — Interest and WorkQueue retention would evict messages and break the running total.
• Nats-Counter-Sources grows — source entries are never removed, even if you later drop the source. That is intentional (it prevents miscounts on re-add), but the header size scales with the number of sources you have ever connected.
• Leaf subjects may disappear after transforms — if count.pl.hits sources into count.eu.hits via subject transform, the original pl subject is not visible at the continental tier. Mirror-then-source if you need both views.
• JavaScript precision — parse counter values as big integers, not numbers. A total over 2^53 will silently corrupt if you parse it as a JS Number.
• Client validation is the client’s job — NewCounterFromStream checks AllowMsgCounter and AllowDirect up front, but if you hand-roll the headers against a non-counter stream, the error surface is less friendly.

Wrapping Up

Distributed counters are the kind of primitive that used to live in a specialist datastore or a custom service. NATS 2.12 turns them into a stream flag. You get order-independent counting across clusters, automatic multi-region aggregation through sources, arbitrary precision, and the same ack-on-publish path you already use — with no extra moving parts in your stack.

For the full design rationale, including revision notes and corner cases, read ADR-49: JetStream Distributed Counter CRDT and the NATS 2.12 release notes.

Requires NATS Server 2.12+ (JetStream API Level 2). See the official documentation for the complete reference.