NATS JetStream: Retrieve data from Streams with Consumers

Liked this content? Check out the related Streams screencast

Or you can Check out the repo and learn at your own pace!

If you’re building distributed systems with NATS, understanding JetStream Consumers is crucial for effectively managing your message streams. In this post, Colin shows us what Consumers are, how they work, and how to implement them in your architecture.

What Are NATS JetStream Consumers?

In the NATS JetStream ecosystem, Consumers are the mechanism through which services retrieve data from Streams. Think of a Stream as persistent storage for your messages on disk, and Consumers as configurable filters that allow services to pull specific subsets of that data.

Here’s the key concept: while Streams store all your messages durably, Consumers let you:

Filter messages by subject
Control where to start reading (from the beginning, from a specific point, or only new messages)
Manage message acknowledgment and redelivery
Load balance work across multiple service instances automatically

The Architecture

In a typical NATS JetStream setup, your architecture might look like this:

Publishers send messages to NATS subjects
Streams persist those messages to disk based on subject filters
Consumers provide filtered access to Stream data
Subscribers pull messages from Consumers for processing

The beauty of this design is that you can have multiple Consumers on a single Stream, each serving different purposes. This enables powerful event-driven patterns like fan-out, fan-in, and saga orchestration.

Types of Consumers: Durable vs Ephemeral

Durable Consumers

Durable Consumers maintain state about which messages have been processed. They’re perfect for:

Long-running services that need to resume from where they left off
Ensuring messages aren’t reprocessed after a service restart
Production workloads requiring guaranteed message delivery

Key characteristics:

Named and persistent
Track acknowledgment state
Survive server restarts
Allow multiple services to share the workload

Ephemeral Consumers

Ephemeral Consumers are temporary and don’t track state. They’re ideal for:

Ad-hoc data analysis
Replaying historical data
Development and debugging
One-time data migrations

Key characteristics:

Auto-generated names
No state persistence
Automatically cleaned up when inactive
Start fresh each time they’re created

Setting Up a Durable Consumer

Colin walks us through setting up a durable Consumer using Infrastructure as Code (OpenTofu/Terraform):

1
resource "nats_consumer" "answers_consumer" {
2
  stream_id     = nats_stream.answers.id
3
  durable_name  = "answers-consumer"
4
  deliver_all   = true
5
  filter_subject = "answers.significant"
6
}

This configuration creates a Consumer that:

Connects to the “answers” Stream
Has a durable name for state persistence
Delivers all historical messages
Only processes messages on the “answers.significant” subject

Implementing Consumer Subscriptions in Python

Here’s how to subscribe to a Consumer in your application code:

1
async def subscribe_and_process(nc, is_long_running=True):
2
    # Subscribe to the durable consumer
3
    subscription = await nc.pull_subscribe(
4
        subject="answers.significant",
5
        durable="answers-consumer"
6
    )
7

8
    while True:
9
        try:
10
            # Fetch messages with a timeout
11
            messages = await subscription.fetch(1, timeout=5)
12

13
            for msg in messages:
14
                # Process the message
15
                await handle_message(msg)
16

17
                # Acknowledge successful processing
18
                await msg.ack()
19

20
        except TimeoutError:
21
            if not is_long_running:
22
                break  # Exit for short-lived processes
23
            continue  # Keep trying for long-running services

Load Balancing Magic

One of NATS JetStream’s most powerful features is automatic load balancing. When you deploy multiple instances of a service that subscribe to the same Consumer, NATS automatically distributes messages among them.

In our example deployment with three pods:

Pod 1 processed 103 messages
Pod 2 processed 102 messages
Pod 3 processed 103 messages

This distribution happens automatically based on each pod’s processing speed and readiness to accept new messages. No additional load balancer needed!

Authorization and Security

Don’t forget to authorize services to access your Consumers. In your NATS account configuration:

1
user {
2
  name = "nats-recorder"
3
  permissions {
4
    subscribe = ["answers.*"]
5
  }
6
}

Without proper authorization, services will be rejected when attempting to subscribe to Consumer subjects.

Creating Ephemeral Consumers

Creating an ephemeral Consumer is even simpler: just subscribe without specifying a durable name:

1
# This creates an ephemeral consumer automatically
2
subscription = await nc.pull_subscribe(
3
    subject="answers.throwaway"
4
)
5

6
# Process messages
7
async for msg in subscription.messages:
8
    await handle_message(msg)
9
    await msg.ack()

Best Practices

Reliability

Use Durable Consumers for Production: They ensure reliable message processing and survive restarts
Acknowledge Messages: Always acknowledge processed messages to prevent redelivery
Set Appropriate Timeouts: When fetching messages, always use timeouts to prevent blocking indefinitely

Performance

Use Subject Filtering: Filter at the Consumer level to reduce unnecessary message processing
Monitor Consumer Lag: Keep track of unprocessed messages to ensure your consumers keep up with the stream

Monitoring Your Consumers

You can monitor Consumer health using the NATS CLI:

# List consumers on a stream
nats consumer ls answers

# Get detailed consumer info
nats consumer info answers answers-consumer

# Check unprocessed messages
nats stream subjects answers

Conclusion

NATS JetStream Consumers provide a powerful abstraction for building resilient, scalable event-driven systems. By understanding the difference between durable and ephemeral consumers, implementing proper authorization, and leveraging automatic load balancing, you can build robust distributed systems that handle millions of messages with ease.

Whether you’re building microservices that need guaranteed message delivery or creating data pipelines that process event streams, NATS JetStream Consumers give you the flexibility and reliability you need.

Ready to dive deeper? Check out the official NATS documentation.