What are Streams in NATS?

Definition

In the NATS ecosystem, streams are the foundational units of persistence within JetStream, NATS’s streaming and storage subsystem. A stream acts as a logical container for messages, storing them durably and organizing them for efficient retrieval, replay, and consumption. Streams allow applications to handle historical and real-time data with advanced features like retention policies, replication, and message filtering.

In the context of NATS Server import and export account configuration, the term stream does not refer to and should not be confused with a JetStream stream (naming things is hard: account import/export predates JetStream). It is just a stream of (Core NATS) messages.

Key Characteristics

Persistent Storage: Streams store messages durably, ensuring data reliability and recovery.
Subject-Based Organization: Messages are written to streams based on their subjects, allowing precise and flexible routing.
Retention Policies: Configurable rules determine how long messages are retained based on time, size, or number of messages.
Replication: Streams can replicate data across clusters for fault tolerance and high availability.
Dynamic Scalability: Streams adapt to workload demands and can integrate seamlessly with additional NATS components like Consumers.

Core Features

Subject Filtering
- Streams capture messages published to specific subjects or patterns, enabling focused and efficient data storage.
Replay Capabilities
- Messages stored in a stream can be replayed by consumers at any time, from the beginning, a specific sequence, or a timestamp.
Retention Options
- Streams support multiple retention modes, such as time-based (e.g., 7 days), size-based (e.g., 1 GB), or interest-based (only for active subscribers).
Replication and High Availability
- Data in a stream can be replicated across NATS clusters to ensure durability and resilience.
Integration with Consumers
- Consumers operate on streams to deliver messages to applications, enabling acknowledgment and fault-tolerant processing.
Mirroring and Sourcing
- Streams can mirror or source data from other streams, creating derived datasets or ensuring redundancy across systems.

Use Cases

Event Sourcing
- Store and replay events for reconstructing system state in event-driven architectures.
Data Streaming
- Handle telemetry, logs, or other time-series data with high-throughput and low-latency requirements.
Audit and Compliance
- Maintain durable logs of system activity or business transactions for auditing or compliance purposes.
Disaster Recovery
- Replicate streams across regions to recover from infrastructure failures and ensure business continuity.
IoT Applications
- Persist telemetry or sensor data from IoT devices, enabling real-time and historical analysis.

Comparison: Streams vs. Traditional Log-Based Storage

Aspect	NATS Streams	Traditional Log-Based Systems (e.g., Kafka)
Setup Complexity	Lightweight, dynamically configurable.	Requires more resources and setup effort.
Data Organization	Subject-based with hierarchical filtering.	Partition-based, fixed during creation.
Retention	Flexible (time, size, interest-based).	Primarily size or time-based.
Replication	Built-in, configurable per stream.	Typically requires external tools.
Edge Compatibility	Optimized for lightweight edge-first use cases.	Heavier, not ideal for constrained environments.
Replay Capabilities	Sequence, timestamp, and real-time replay.	Offset-based replay with limited flexibility.

Associated Components and How They Interoperate

JetStream:
- Streams are a core feature of JetStream, providing the storage layer for durable and replayable messaging.
Consumers:
- Operate on streams to deliver messages, with support for filtering, acknowledgment, and replay.
Subjects and Wildcards:
- Streams use subjects to filter messages, enabling flexible organization and efficient retrieval.
Mirroring and Sourcing:
- Extend streams across clusters or derive new datasets by mirroring or sourcing existing streams.
Monitoring Tools:
- Integrate with observability systems like Prometheus and Grafana to monitor stream performance, retention, and message delivery.

Additional Resources

Streams in NATS JetStream empower developers to build robust, fault-tolerant, and scalable applications. By providing durable storage, advanced replay capabilities, and seamless integration with consumers and other NATS components, streams are an essential tool for modern, distributed messaging systems.