How to Connect NATS.io to Apache Kafka with Redpanda Connect

You can connect Apache Kafka to NATS by using Redpanda Connect as a bridge: Kafka acts as the input, Redpanda Connect optionally transforms each event, and NATS JetStream receives the processed messages as the output. This tutorial walks through a local Kafka-to-NATS integration using a Kafka topic, a Redpanda Connect YAML configuration, and a NATS JetStream subject.

By the end, you will have messages flowing from a Kafka topic named kafka-nats into NATS JetStream through Redpanda Connect.

What are you building?

This tutorial builds a simple Kafka-to-NATS bridge:

• Apache Kafka produces events to a topic.
• Redpanda Connect, formerly known as Benthos, consumes those events, applies a small transformation, and forwards them.
• NATS receives the processed events through JetStream.

The flow looks like this:

When should you connect NATS and Kafka?

Connecting NATS and Kafka is useful when you need to move event streams between systems with different operational models or integration needs. Common scenarios include:

• Bridging existing Kafka pipelines to NATS-based services
• Distributing Kafka events to applications running across cloud, on-premises, or edge environments
• Applying lightweight filtering, transformation, or enrichment before messages enter NATS
• Supporting replication, backup, or real-time analytics workflows that use multiple sources and sinks

NATS and Kafka overlap in some messaging and streaming capabilities, but they are often used for different parts of an architecture. Kafka is commonly used as a distributed event streaming platform for durable logs, data pipelines, and analytics. NATS provides a cloud-native connective fabric with subject-based messaging, request-reply, publish-subscribe, streams, object stores, and JetStream persistence.

How do NATS and Kafka differ in this integration?

NATS uses subject-based messaging. Publishers send messages to hierarchical string values called subjects, and subscribers listen on subjects or subject patterns. Kafka uses topics and partitions as the primary abstraction for organizing event streams.

NATS JetStream provides persistence for NATS messages and supports file- and memory-based storage. Kafka provides log-based storage for topics. In this tutorial, Kafka is the source topic and NATS JetStream is the destination for processed messages.

Prerequisites

Before you start, install or have access to:

• Apache Kafka
• The rpk client
• A local terminal environment where you can run Kafka, NATS, and Redpanda Connect commands

This tutorial uses a local, self-hosted NATS server. For a managed NATS environment, you can use Synadia Cloud, which provides a managed global NATS service with infrastructure, account and user management, and JetStream management.

Step 1: How do you set up NATS with JetStream?

This tutorial uses two separate NATS programs, distributed independently:

• The nats CLI, used to create and inspect streams and to run connection checks.
• The nats-server binary, which runs the NATS server itself.

Install the nats CLI with the official installer:

curl -sf https://binaries.nats.dev/nats-io/natscli/nats | sh

Install the nats-server binary separately. On macOS you can use Homebrew:

brew install nats-server

For other platforms, download the release archive that matches your operating system and CPU architecture from the nats-server releases page, then place the extracted nats-server binary on your PATH or in your working directory.

Start the NATS server with JetStream enabled

Start the NATS server with JetStream enabled and set the JetStream storage directory to the current folder:

nats-server -js --store_dir .

If you placed the nats-server binary in the current directory instead of on your PATH, run it as ./nats-server -js --store_dir ..

Keep this terminal window open. In a new terminal window, run a connection check:

nats server check connection -s nats://127.0.0.1:4222

This validates that your local NATS server is reachable before you configure Redpanda Connect to publish to it.

Create the JetStream stream that stores the bridged messages

JetStream persists a message only if a stream is configured to capture the subject the message is published on. Redpanda Connect publishes to the NATS subject kafka-nats-subject (configured in Step 4), so you must create a stream that captures that subject before you start the bridge. If no stream captures the subject, the JetStream publish from Redpanda Connect fails and no messages are stored.

Create a stream named kafka-nats that captures the kafka-nats-subject subject:

nats stream add kafka-nats --subjects "kafka-nats-subject" --storage memory --defaults

The --defaults flag accepts the default values for all other stream options; omit it to be prompted for each option interactively instead. This example uses memory storage, so the stream’s messages are cleared when the server restarts. Use --storage file if you want them persisted under the --store_dir location.

Step 2: How do you create the Kafka topic?

The kafka-*.sh helper scripts used below ship in the bin/ directory of your Apache Kafka installation, and server.properties is the broker configuration file in its config/ directory. Run these commands from the directory where you extracted Kafka, and adjust the paths if your layout differs.

Start the Kafka server by pointing it to the broker configuration file:

bin/kafka-server-start.sh config/server.properties

After Kafka starts, create a topic named kafka-nats:

bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic kafka-nats

This topic is the source that Redpanda Connect will consume from.

Optional: How do you validate Kafka before connecting it to NATS?

You can create a Kafka console consumer to verify that publishing and consumption work inside Kafka before you add NATS to the flow:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic kafka-nats --from-beginning

This step is optional. Once Kafka validation is complete, Redpanda Connect will consume messages from the same topic and forward them to NATS.

Step 3: How do you install Redpanda Connect?

Download Redpanda Connect with curl:

curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-darwin-arm64.zip

IMPORTANT: The download artifact can differ by operating system and CPU architecture. Use the Redpanda Connect download that matches your environment.

Create a local binary directory, add it to your PATH, and unzip the rpk command line client:

mkdir -p ~/.local/bin
export PATH=$PATH:~/.local/bin
unzip rpk-darwin-arm64.zip -d ~/.local/bin/

Confirm that the installation succeeded:

rpk --version

Step 4: How do you configure Redpanda Connect to read from Kafka and write to NATS JetStream?

Create a Redpanda Connect configuration file named connect.yaml. In this configuration:

• Kafka is the input.
• The pipeline transforms each JSON message.
• NATS JetStream is the output.

1
input:
2
  kafka:
3
    addresses:
4
      - localhost:9092
5
    topics:
6
      - kafka-nats
7
    consumer_group: kafka-nats-group
8

9
pipeline:
10
  processors:
11
    - sleep:
12
        duration: 50ms
13
    - mapping: |
14
        root.company = this.data.company.name
15
        root.name = this.data.company.contactperson.name
16
        root.email = this.data.company.contactperson.email.hash("sha256")
17
        root.orderdate = this.data.dateoforder
18

19
output:
20
  nats_jetstream:
21
    urls: [ "nats://127.0.0.1:4222" ]
22
    subject: "kafka-nats-subject"

What does the Kafka input configuration do?

The input.kafka section tells Redpanda Connect where to read events from:

• addresses: The Kafka bootstrap server. This example uses localhost:9092.
• topics: The topic or topics to consume. This example uses kafka-nats.
• consumer_group: The Kafka consumer group. This example uses kafka-nats-group.

These fields identify the Kafka source for the bridge.

What does the pipeline transform?

The pipeline section extracts fields from the nested source JSON and writes a flatter output message:

• company: The company name
• name: The name of the person who placed the order
• email: A SHA-256 hash of the contact person’s email address
• orderdate: The order date

This shows how Redpanda Connect can process messages while moving them between Kafka and NATS.

What does the NATS JetStream output configuration do?

The output.nats_jetstream section tells Redpanda Connect where to publish processed messages:

• urls: The NATS server URL. This tutorial uses nats://127.0.0.1:4222.
• subject: The NATS subject used for the published messages. This example uses kafka-nats-subject.

Step 5: How do you start the Kafka-to-NATS bridge?

Run Redpanda Connect with the configuration file:

rpk connect -c connect.yaml

You should see output similar to this:

INFO Running main config from specified file       @service=benthos benthos_version=24.1.9 path=connect.yaml
INFO Launching a Redpanda Connect instance, use CTRL+C to close  @service=benthos
INFO Output type nats_jetstream is now active      @service=benthos label="" path=root.output
INFO Input type kafka is now active                @service=benthos label="" path=root.input

This confirms that Redpanda Connect has started the Kafka input and the NATS JetStream output.

Step 6: How do you publish test messages to Kafka?

Publish each object below as an individual JSON message to the kafka-nats Kafka topic. The source messages use a nested structure and include an unhashed email address.

Open a Kafka console producer in a new terminal:

bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic kafka-nats

Paste each JSON object below on its own line. The console producer sends every line as a separate Kafka message, and the Redpanda Connect mapping expects each message to contain a single data object.

1
{"data":{"company":{"name":"JD Inc.","contactperson":{"name":"John Doe","email":"john.doe@jdmail.com"}},"price":"1050.00","dateoforder":"12/07/2024"}}
2
{"data":{"company":{"name":"Smith Enterprises","contactperson":{"name":"Jane Smith","email":"jane.smith@semail.com"}},"price":"1200.00","dateoforder":"15/07/2024"}}
3
{"data":{"company":{"name":"Tech Solutions","contactperson":{"name":"Mark Brown","email":"mark.brown@tsmail.com"}},"price":"980.00","dateoforder":"18/07/2024"}}
4
{"data":{"company":{"name":"Innovate LLC","contactperson":{"name":"Lucy Green","email":"lucy.green@illcmail.com"}},"price":"1150.00","dateoforder":"20/07/2024"}}
5
{"data":{"company":{"name":"Global Corp","contactperson":{"name":"Michael White","email":"michael.white@gcmail.com"}},"price":"1300.00","dateoforder":"22/07/2024"}}
6
{"data":{"company":{"name":"Future Tech","contactperson":{"name":"Laura Blue","email":"laura.blue@ftmail.com"}},"price":"1075.00","dateoforder":"25/07/2024"}}
7
{"data":{"company":{"name":"Bright Ideas","contactperson":{"name":"David Black","email":"david.black@bimail.com"}},"price":"1250.00","dateoforder":"28/07/2024"}}
8
{"data":{"company":{"name":"NextGen Inc.","contactperson":{"name":"Olivia Gray","email":"olivia.gray@ngmail.com"}},"price":"1100.00","dateoforder":"30/07/2024"}}
9
{"data":{"company":{"name":"Prime Solutions","contactperson":{"name":"Paul Brown","email":"paul.brown@psmail.com"}},"price":"995.00","dateoforder":"02/08/2024"}}
10
{"data":{"company":{"name":"Smart Innovations","contactperson":{"name":"Emily White","email":"emily.white@simail.com"}},"price":"1400.00","dateoforder":"05/08/2024"}}
11
{"data":{"company":{"name":"Elite Enterprises","contactperson":{"name":"James Black","email":"james.black@eemail.com"}},"price":"1180.00","dateoforder":"08/08/2024"}}

Step 7: How do you validate that messages reached NATS?

After publishing messages to Kafka, inspect the NATS stream data:

nats stream view kafka-nats

You can also view a stream summary:

nats stream report

Example output:

Obtaining Stream stats

╭───────────────────────────────────────────────────────────────────────────────────────────────╮
│                                         Stream Report                                         │
├────────────┬─────────┬───────────┬───────────┬──────────┬─────────┬──────┬─────────┬──────────┤
│ Stream     │ Storage │ Placement │ Consumers │ Messages │ Bytes   │ Lost │ Deleted │ Replicas │
├────────────┼─────────┼───────────┼───────────┼──────────┼─────────┼──────┼─────────┼──────────┤
│ kafka-nats │ Memory  │           │         0 │ 11       │ 1.7 KiB │ 0    │       0 │          │
╰────────────┴─────────┴───────────┴───────────┴──────────┴─────────┴──────┴─────────┴──────────╯

The report confirms 11 messages were consumed from Kafka and published to NATS JetStream.

If the Redpanda Connect mapping is working correctly, the transformed messages stored in NATS look similar to the following (the first 10 of the 11 messages are shown):

1
[
2
  {"company":"JD Inc.","email":"/y2xKnsBxZ9svYXLJYreF+KvZcGz30sSCii0dOt5RHs=","name":"John Doe","orderdate":"12/07/2024"},
3
  {"company":"Smith Enterprises","email":"J/XjAZzFMk1wE5b4m/G/T5uOAeRd6ep/TNUw51RA8Rs=","name":"Jane Smith","orderdate":"15/07/2024"},
4
  {"company":"Tech Solutions","email":"HFxJTvcjZ+Oq4VEAn2In9doLTGTekKIlA4F9WmSceAY=","name":"Mark Brown","orderdate":"18/07/2024"},
5
  {"company":"Innovate LLC","email":"g2Kov5zhGrqBF6D4DnX/vG6qkby5w1cpvGO9W0ZVHow=","name":"Lucy Green","orderdate":"20/07/2024"},
6
  {"company":"Global Corp","email":"kTintSH97blGqknuTP4mJahDEdYR0upEqBs7/sS+Coc=","name":"Michael White","orderdate":"22/07/2024"},
7
  {"company":"Future Tech","email":"cF2sFbDnLWrWwg7LHedliIU4MFBSTBtT2MItO3/FXxQ=","name":"Laura Blue","orderdate":"25/07/2024"},
8
  {"company":"Bright Ideas","email":"nR80FyCZTH/hvNb+7v5Nj2l9ShrNuWnjMdh+omhAHuU=","name":"David Black","orderdate":"28/07/2024"},
9
  {"company":"NextGen Inc.","email":"lG+gaAuCan4yptV7AFNe5k+Pfne1+qunin4BGN6k38M=","name":"Olivia Gray","orderdate":"30/07/2024"},
10
  {"company":"Prime Solutions","email":"jMqGCWexvhOMRGfYQJE0VrQE6uzVNjT1LhGA+EYNziM=","name":"Paul Brown","orderdate":"02/08/2024"},
11
  {"company":"Smart Innovations","email":"imyi4RNgnsQLs1T2HyhWM58cGhEL+Sg9eP6Sqaf10h8=","name":"Emily White","orderdate":"05/08/2024"}
12
]

Messages from Apache Kafka can now flow to NATS through the Redpanda Connect pipeline.

If you need the commands, configuration files, or sample data for reference, they are available in this GitHub repository.

Troubleshooting common Kafka-to-NATS connection issues

Why is Redpanda Connect not consuming from Kafka?

Check that Kafka is running, the kafka-nats topic exists, and the addresses value in connect.yaml matches your Kafka bootstrap server. For this tutorial, the expected value is localhost:9092.

Why is Redpanda Connect not publishing to NATS?

Confirm that the NATS server is still running with JetStream enabled and that the urls value in connect.yaml matches the NATS server address. In this tutorial, the expected URL is nats://127.0.0.1:4222.

Also confirm that a JetStream stream capturing the kafka-nats-subject subject exists. The nats_jetstream output publishes through JetStream, and JetStream stores a message only when a stream is configured to capture its subject. If that stream is missing, the publish fails and no messages reach NATS. Run nats stream report to confirm the kafka-nats stream is present, or create it as shown in Step 1.

Why do the output messages look different from the input messages?

The Redpanda Connect mapping processor reshapes each source event. It extracts selected nested fields, hashes the email value with SHA-256, and writes a flatter JSON object to NATS.

Why should each test object be sent as an individual message?

The pipeline mapping expects each message to contain a data object with the nested structure shown in the sample payload. Sending each object as its own event makes the transformation apply to each order independently.

Next steps

In this tutorial, you connected Apache Kafka to NATS using Redpanda Connect. Kafka served as the source system, Redpanda Connect acted as the bridge and stream processing layer, and NATS JetStream received the processed messages.

NATS can serve as a connective fabric for applications and data flows across cloud, on-premises, and edge environments. For a managed NATS option, Synadia Cloud provides a managed global service for NATS with infrastructure, account and user management, and JetStream management.

For more practical examples, visit NATS by Example. To avoid managing NATS infrastructure yourself, you can sign up for a free Synadia Cloud account and start building with NATS.

Want help from the NATS experts? Meet with our architects to get help tailored to your use case and environment.