Do You Really Need MQTT and Kafka for Industrial IoT?

The short answer

The standard industrial IoT stack bridges an MQTT broker into Apache Kafka: MQTT owns the devices, Kafka owns durability, and a connector glues them together. Both halves are good at their jobs. The connector is where you pay: two distributed systems to run, two security and monitoring models, and a bridge that is itself a failure point in the middle of your most important data path. The fix is a fabric where the two jobs were never split, not a second proprietary broker. Open-source NATS (Apache-2.0) terminates MQTT 3.1.1 directly in the server, keeps device sessions and quality-of-service (QoS) state in JetStream streams, and those same streams provide end-to-end acknowledgments, bounded redelivery, a dead-letter queue (DLQ) pattern, and replay by time or sequence, from a single binary that runs identically at the plant edge and in the cloud. If your analytics estate runs on Kafka, keep it: sink from NATS into it. The operational path simply no longer needs two brokers and a bridge to be trustworthy.

The comparison is simple to picture. On the left, devices connect to an MQTT broker, a connector copies data into Kafka, and Kafka feeds the cloud: three moving parts on the critical path from device to decision. On the right, one fabric spans device, edge, and cloud, and there is no bridge in the middle.

Why do industrial IoT architectures use both MQTT and Kafka?

Start by giving the stack its due, because it is everywhere for good reasons.

MQTT is the right protocol at the device edge. Programmable logic controllers, gateways, sensors, and vision systems are constrained clients on flaky last-hop links. MQTT’s small footprint, persistent sessions, and QoS levels are a genuinely good fit for that environment. If a wireless link drops mid-shift, MQTT is designed to pick the session back up. There is nothing wrong with MQTT at the edge, and this post does not argue that there is.

Behind the devices, though, the requirement changes. A plant needs durable, replayable history: the ability to rebuild a dashboard’s state after a restart, to reprocess a shift’s worth of quality events through a new model, to answer “what did line 3 actually report between 02:00 and 03:00” days later. That is a streaming requirement, and MQTT brokers were not built to satisfy it. MQTT brokers do persist session state, but session persistence is not the same thing as a replayable log of everything that happened.

Kafka was the available answer to that second requirement. It is a durable, partitioned log with a large stream-processing ecosystem, and it does that job well. So the industry did the rational thing: use MQTT for what MQTT is good at, use Kafka for what Kafka is good at, and connect them. The result, an MQTT broker bridged into Kafka, is not a mistake. It is a reasonable composition of two strong tools. The problem is what it costs to keep those two tools in step.

What are the costs of bridging an MQTT broker into Kafka?

The costs are structural, not incidental. They do not come from picking the wrong MQTT broker or the wrong Kafka distribution. They come from running two distributed systems with a connector between them.

Two of everything operational. Two systems means two upgrade cycles, two monitoring stacks, two sets of tuning knobs, two failure modes to learn, and on-call coverage for both. When something is slow on the path from device to decision, the first question is always “which side?”, and answering it means correlating across two systems that keep their metrics and logs in different places.

The connector is a failure point on your most important data path. The bridge that copies data from MQTT into Kafka is itself a distributed component. It has its own liveness, its own backpressure behavior, its own delivery semantics, and its own latency step. When it stalls, devices are still publishing and consumers are still reading, but the two halves have quietly diverged. A work-order event acknowledged at the MQTT broker has not necessarily landed durably in Kafka yet, and that gap stays invisible until an audit or an incident forces you to look.

A data-model impedance mismatch. MQTT is a hierarchical topic tree; Kafka is a partitioned log. The connector has to map one onto the other, and that mapping is a design decision you now own and maintain: how topics become topics-or-partitions, how ordering is preserved, how keys are assigned. Nobody did anything wrong here; the two systems model data differently, and something in the middle has to reconcile that on every message.

Two security models. Device authentication, authorization, and transport security at the MQTT broker are configured and audited separately from the equivalents in Kafka. Every identity, every access rule, and every certificate rotation has to be reasoned about twice, and the seam between the two models is where mistakes hide.

None of this is an attack on MQTT brokers or on Kafka; it is the cost of the seam between them.

Is a unified broker better than MQTT plus Kafka?

There is an emerging answer to this pain: consolidate device connectivity and durable streaming onto a single proprietary broker and delete the bridge. That is a real improvement over the two-broker status quo. Removing the connector removes a failure point, a latency step, and a whole class of divergence bugs. Collapsing the seam is the correct instinct.

The question is what you take on in exchange. Consolidating onto a single proprietary broker puts the entire nervous system of the plant, every device session and every durable stream, behind one commercial license and one roadmap you do not control. That is license coupling on the most load-bearing part of your architecture, and it is a procurement dependency rather than an engineering one. And in practice the second system often does not fully leave: very-large-scale retention and mature stream processing are frequently still delegated to Kafka behind the unified broker. The seam has not disappeared; it has moved somewhere you can no longer see it.

Trading an integration problem for a procurement problem is a lateral move. The better question is whether the two jobs have to be split at all.

What happens to messages when a consumer is offline or fails?

This is the real test: any replacement for the two-broker stack has to answer a specific set of reliability questions before you can trust it on the operational path. Here are the ones that matter, mapped to the open-source NATS primitive that answers each. Hold any candidate against this checklist, not just NATS.

Concretely: a quality-rejection event published while the analytics consumer is redeploying is not lost, because the durable consumer resumes from its last acknowledged message when it comes back. A device command that must not be silently dropped can wait on a publish acknowledgment that confirms the command was persisted, and on a request/reply exchange that confirms it was executed. A shift’s worth of events can be replayed through a new model by starting a fresh consumer at a timestamp.

On dead-letter queues. NATS has no single-config, checkbox DLQ. The pattern is explicit: you bound redelivery with MaxDeliver; when a message exceeds that bound the server publishes an advisory on $JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.<stream>.<consumer> (terminated messages advise on $JS.EVENT.ADVISORY.CONSUMER.MSG_TERMINATED.<stream>.<consumer>, and negative acknowledgments on $JS.EVENT.ADVISORY.CONSUMER.MSG_NAKED); you capture those advisories into their own stream; and a small handler fetches the original message by stream sequence with a direct get and republishes it to a DLQ subject. That is more assembly than flipping a config flag. In return, the resulting dead-letter stream is a first-class, replayable stream with its own retention, filtering, and replay, not a black box you can only drain by hand. The mechanics of building this end to end are covered in the companion builder guide on reliable message delivery in JetStream, and the broader argument for putting these guarantees in the infrastructure rather than in application code is the subject of the companion concept post in this pack.

Can one system handle both device connectivity and durable streams?

This is where the two-broker assumption breaks. The reasonable objection is: “If I remove the MQTT broker, how do my devices connect?”

They connect to the NATS server, because to an MQTT device the NATS server is an MQTT broker. The server has an MQTT 3.1.1 listener built in. Devices keep their existing client libraries and connection code; nothing on the device changes. Under the hood, MQTT sessions and QoS state are persisted in JetStream streams, the same durable substrate that gives you replay. Device connectivity and durable streaming are one system seen from two angles, not two systems bridged together.

Enabling the listener is a small addition to nats-server.conf; MQTT requires JetStream to be enabled, since that is where session state lives:

1
jetstream {
2
  store_dir: "/data/jetstream"
3
}
4

5
mqtt {
6
  port: 1883
7
}

The limits, stated up front:

• The listener accepts QoS 0, 1, and 2, with configuration options to reject or downgrade QoS 2 if you prefer.
• The protocol level is MQTT 3.1.1 only, not 5.0, and 5.0 is not planned. If you specifically require MQTT 5.0 features at the device, this is not the right fit.
• NATS carries Sparkplug B payloads faithfully but does not interpret Sparkplug semantics.

Because it is a single binary with the same feature set at the edge and in the cloud, the same server that terminates MQTT on a plant-floor gateway is the same server that runs your durable streams in the data center. Leaf nodes let the edge keep operating when the wide-area link is down and reconcile when it returns.

Do I lose Kafka’s analytics ecosystem if I consolidate?

No. Kafka is genuinely strong at very-large-scale retention and has a deep, mature stream-processing ecosystem. If you have already built an analytics estate on Kafka, there is no reason to tear it out, and this post is not asking you to.

The claim here is narrower and, for that reason, stronger. Nobody is telling you to replace Kafka everywhere. The operational path, device to decision, the part that has to be trustworthy in real time, should be one system rather than two brokers and a bridge. Keep Kafka where it earns its place in analytics, and sink into it from NATS. You get the ecosystem you have invested in without a connector sitting on the critical path between your devices and your control logic.

Landscape

Here is how the options compare. Where a capability genuinely varies by product, the table says so rather than overclaiming.

FAQ

Why do industrial IoT architectures use both MQTT and Kafka? MQTT is an excellent fit at the device edge: small, session-aware, and tolerant of flaky links. Durable, replayable history behind the devices is a separate requirement, and Kafka was the available tool for it. The two-broker stack is a rational composition; the cost is the connector that keeps them in step.

What are the costs of bridging an MQTT broker into Kafka? You run two distributed systems with two upgrade cycles, two monitoring stacks, and two security models. The connector is itself a failure point and a latency step on your most important data path, and it has to reconcile MQTT’s topic tree with Kafka’s partitioned log on every message.

Is a unified broker better than MQTT plus Kafka? Consolidating onto one broker does remove the bridge, which is a real improvement. Doing it with a proprietary broker trades an integration problem for a procurement problem: license coupling on the whole plant nervous system, plus analytics often still delegated to Kafka. A single open fabric removes the bridge without the license coupling.

Can one system handle both device connectivity and durable streams? Yes. Open-source NATS terminates MQTT 3.1.1 directly in the server and persists device sessions in the same JetStream streams that provide durable, replayable history. Device connectivity and durable streaming are one system, not two bridged together.

What happens to messages when a consumer is offline or fails? Durable streams and durable consumers hold the data; when a consumer returns it resumes from its last acknowledged message. Consumers ack, nak (redeliver, optionally with backoff), or term (stop trying), and MaxDeliver bounds retries so a poison message ends in a dead-letter pattern rather than an infinite loop.

Do I lose Kafka’s analytics ecosystem if I consolidate? No. Kafka remains strong for very-large-scale retention and stream processing. Keep that estate and sink into it from NATS. What changes is that the operational path from device to decision no longer needs two brokers and a bridge to be trustworthy.

Go deeper

• JetStream, consumers, acknowledgments, and replay policies: docs.nats.io
• The built-in MQTT listener and its configuration: docs.nats.io
• Architecture Decision Records for JetStream and consumer semantics: the NATS ADR repository on GitHub
• Runnable, copy-pasteable patterns across client languages: natsbyexample.com
• Server source and clients: the NATS GitHub organization

The snippet above is server configuration; for consumers and handlers, the Go client is the usual starting point, with equivalent clients in other languages. For the end-to-end mechanics of acks, retries, dead-letter handling, and replay, see the companion builder guide in this pack; for why these guarantees belong in the infrastructure rather than in application code, see the companion concept post.

Want help from the NATS experts? Meet with our architects to get help tailored to your use case and environment.