Message Reliability Belongs in the Infrastructure, Not in Your Application Code
The short answer
If your services are full of retry loops, outbox tables, and deduplication caches, your messaging layer is exporting its reliability problems to every team that builds on it. Transport-level reliability (surviving offline consumers, retrying failures with backoff, quarantining poison messages, replaying history) belongs in the infrastructure, implemented once. Five questions tell you whether yours does it: What happens when the consumer is offline? When it rejects a message? When a downstream system is out for hours? When you need to rebuild state from history? When a command must be confirmed? But not everything moves down: idempotency and semantic validation stay with the application, because only the application knows what a message means. Infrastructure owns delivery; applications own meaning.
The shadow reliability layer
Walk through most distributed systems and you will find the same set of defensive structures repeated in service after service: a retry loop with a hand-tuned backoff, an outbox table that stages events until a background worker can push them, a deduplication cache keyed on some message ID, a nightly re-sync job, a “reconciliation” service whose entire job is to detect and repair drift between two systems that were supposed to stay in sync.
None of these are features. Each one exists because the messaging layer underneath guaranteed too little, and an application team was left to rebuild the missing reliability by hand, differently from the team next door and with a different set of bugs.
Consider a factory floor. A manufacturing execution system raises a quality-rejection event: a batch has failed inspection and must be held. That event needs to reach the ERP system so the batch is flagged before it ships, the maintenance system so the line can be inspected, and a reporting pipeline for the compliance record. The publisher fires the event and moves on. But the ERP integration happened to be mid-restart, the message hit a link that only guaranteed delivery to the nearest broker, and nobody was subscribed at that instant to catch it. The event is gone. The batch ships. The gap surfaces days later as a manual reconciliation ticket, because someone, somewhere, built a reconciliation service for precisely the failures the messaging layer could not prevent.
Every structure in that shadow layer is compensating for a guarantee the infrastructure didn’t make. The question worth asking is not “how do we build better retry logic” but “why is retry logic living in application code at all.”
Should message reliability live in the application or the infrastructure?
Transport-level reliability should live in the infrastructure, implemented once. Delivering a message to an offline consumer when it returns, retrying a failed handler with backoff, bounding retries so a poison message doesn’t loop forever, holding history so state can be rebuilt: these are generic problems with generic solutions. Solving them once in the fabric is strictly better than solving them N times, badly, in N services.
The part that must not move down is meaning. Whether a message is a duplicate that should be ignored, whether its contents are semantically valid, what business action compensates for a failure that has already been committed: only the application knows any of that. The useful line is transport versus meaning. The infrastructure owns getting the message there reliably; the application owns deciding what the message is and what to do about it.
Why isn’t MQTT QoS an end-to-end guarantee?
MQTT is excellent right at the device edge: lightweight, battery-friendly, and well suited to constrained networks. Its quality-of-service (QoS) levels are real and useful. But a QoS level describes a hop: it governs the link between a client and the broker it is connected to. QoS 1 means the broker acknowledges that it received your publish; QoS 2 adds a handshake so that hop delivers exactly once. That is hop-by-hop reliability, and it is standard protocol behavior.
What it does not describe is the rest of the journey, from the broker to the consumer and through the consumer’s successful processing of the message. A publish can be acknowledged at QoS 2 and still never be acted upon, because the consumer was offline and the broker retained only session state rather than a replayable history of what was published while it was gone. The accurate distinction is session state versus replayable history: MQTT brokers persist sessions and can queue for known clients, but that is a different thing from an append-only log any authorized consumer can rewind.
End-to-end reliability needs two acknowledgments working together: a persisted-publish acknowledgment that confirms the message is durably stored, and a consumer-processing acknowledgment that confirms the message was handled. QoS covers the first hop; the pair of acknowledgments covers the whole journey.
What must a messaging layer guarantee? The five questions
Here is a vendor-neutral way to evaluate any messaging layer, whether that is an MQTT broker, a log-based streaming platform, or a proprietary unified broker. Ask five questions, and insist on a concrete answer to each.
- What happens when the consumer is offline at publish time? A message published while a consumer is down must still be delivered when it returns, not dropped and not dependent on the consumer having registered a session first.
- What happens when a consumer cannot process a message? The consumer needs a way to say done, try me again, and stop trying, and the layer needs to bound retries so a message that can never succeed doesn’t loop forever.
- What happens when a downstream system is out for hours? Redelivery must be bounded and paced, and the consumer must be able to resume exactly where it left off when the system comes back, without a flood or a gap.
- What happens when you need to rebuild state from history? You must be able to start a consumer from an earlier point, a sequence or a timestamp, and replay past messages to reconstruct state.
- What happens when a command needs confirmed delivery? For an actuator command or a state change, you need proof it was persisted, and often proof it was executed. Fire-and-forget is not enough.
If the honest answer to any of these is “the application handles that,” you have found a piece of your shadow reliability layer.
What answering all five looks like
Open-source NATS with JetStream (Apache-2.0) is a useful worked example, because it answers all five in the fabric rather than in your code.
Messages are published into durable streams, and consumers read through a durable cursor. A consumer that was offline at publish time resumes from its last acknowledged message: question one, handled by the infrastructure.
When a consumer receives a message it has three verbs: ack (done), nak (redeliver, optionally after a delay or on a backoff schedule), and term (stop trying). Redelivery is bounded by AckWait and MaxDeliver, so a downstream outage produces paced retries and a clean resume from the cursor rather than a lost message or an infinite loop: questions two and three.
Poison messages deserve a straight answer: NATS has no single-config dead-letter queue. The pattern is explicit: bound redelivery with MaxDeliver; when a message exceeds it, the server publishes an advisory on $JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.<stream>.<consumer> (terminated messages advise on MSG_TERMINATED, naks on MSG_NAKED); a small handler captures those advisories into their own stream, fetches the original message by sequence with a direct get, and republishes it to a dead-letter subject. That is more assembly than a checkbox. In return, the dead-letter queue (DLQ) is a first-class stream with its own retention, filtering, and replay, which is exactly what you want when you eventually need to inspect and reprocess what failed.
For rebuilding state, a consumer’s start position is set by sequence or timestamp, and ReplayPolicy can replay instantly or in original timing. Any authorized consumer can rewind the stream: question four. And for confirmed commands, the publish acknowledgment confirms persistence while request/reply confirms execution: question five.
One fabric, open source, running from the edge to the cloud through leaf nodes and accounts. The two companion posts in this pack go deeper: an architecture piece on whether industrial IoT really needs separate protocols for the edge and the core, and a builder’s guide to acks, retries, dead letters, and replay in JetStream. This is the frame they build on.
What reliability work should stay in the application?
Moving transport reliability down does not empty your application of responsibility. Three things stay put, because they depend on meaning:
- Idempotent handlers. At-least-once delivery is what any durable system can truthfully promise, which means a handler can see the same message twice. Making the effect of processing safe to repeat, through an idempotency key, a conditional write, or a natural upsert, is the application’s job, because only the application knows what “the same” means and what “already applied” looks like.
- Semantic validation. The infrastructure can guarantee a message arrived intact. It cannot tell you that a quality-rejection event references a batch that exists, or that a command’s parameters are within safe range. That is domain knowledge.
- Business-level compensation. When something fails after it has already had an effect, undoing it is a business decision: issue a credit, cancel a hold, reverse a transaction. The fabric can redeliver; it cannot decide what “undo” means.
Reliability is shared. The infrastructure owns delivery; the application owns meaning. Committing to that line is what keeps this an engineering principle rather than a claim that the messaging layer solves everything.
The reliability checklist
| Failure scenario | What good looks like | Where it lives |
|---|---|---|
| Consumer offline at publish time | Durable history; delivery resumes from the last acknowledged message when the consumer returns | Infrastructure |
| Consumer cannot process a message | Three verbs: ack (done), nak (redeliver, optionally with delay/backoff), term (stop trying); retries bounded so poison messages route to a replayable DLQ | Infrastructure |
| Downstream system out for hours | Bounded, paced redelivery (AckWait, MaxDeliver, backoff); consumer resumes from its cursor when the system returns | Infrastructure |
| Rebuild state from history | Start position by sequence or timestamp; replay instant or in original timing; any authorized consumer can rewind | Infrastructure |
| Command needs confirmed delivery | Publish acknowledgment confirms persistence; request/reply confirms execution | Infrastructure |
| Same message processed twice | Handler effect is safe to repeat (idempotency key, conditional write, upsert) | Application |
| Message is well-formed but wrong | Domain validation against business rules and referenced state | Application |
| A committed action must be undone | Business-level compensation (credit, cancel, reverse) | Application |
FAQ
Should message reliability live in the application or the infrastructure? Transport-level reliability (surviving offline consumers, retrying with backoff, bounding poison messages, and replaying history) belongs in the infrastructure, implemented once, so every team inherits it. Idempotency, semantic validation, and business compensation stay in the application, because they depend on what a message means. Infrastructure owns delivery; applications own meaning.
Why isn’t MQTT QoS an end-to-end guarantee? QoS describes a single hop: the link between a client and its broker. It confirms the broker received the publish, but it says nothing about whether the message reached the consumer and was processed successfully. End-to-end reliability needs a persisted-publish acknowledgment plus a consumer-processing acknowledgment, which is a broader guarantee than any per-hop QoS level provides. MQTT remains an excellent choice at the device edge; this is about scope, not a flaw.
What is a dead-letter queue and why does it matter? A dead-letter queue (DLQ) is where messages go after they have exhausted their redelivery attempts, so a message that can never be processed stops blocking or looping and is instead set aside for inspection and reprocessing. It matters because without one, a single poison message can stall a consumer or be silently dropped. In NATS this is a pattern rather than a single config: bound redelivery, capture the resulting advisories into a stream, and republish failed messages onto a dead-letter subject, giving you a first-class, replayable stream with its own retention and filtering.
What does message replay give you? Message replay lets an authorized consumer start from an earlier point in the stream, a sequence number or a timestamp, and re-read past messages to reconstruct state, seed a new service, test against real history, or recover from a bug. Replay can run instantly or preserve the original timing of the messages. The messaging layer’s history becomes a durable source of truth rather than a transient buffer.
What is the difference between at-least-once and exactly-once delivery? At-least-once means a message will be delivered until it is acknowledged, which can mean the same message is seen more than once. Exactly-once end to end is difficult to guarantee across independent systems, so the dependable approach is at-least-once delivery paired with idempotent handling in the application. The infrastructure guarantees the message is not lost; the application guarantees processing it twice has the same effect as processing it once.
Go deeper
- JetStream streams and consumers, acks/naks/term,
MaxDeliver, advisories, andDeliverPolicy/ReplayPolicyin the NATS documentation. - Runnable patterns at natsbyexample.com.
- The NATS server source and ADRs for the exact advisory subjects and delivery semantics.
Two companion posts build directly on this frame: an architecture piece on whether industrial IoT really needs separate protocols at the edge and the core, and a builder’s guide to reliable delivery in JetStream covering acks, retries, dead letters, and replay.
Want help from the NATS experts? Meet with our architects to get help tailored to your use case and environment.
Related posts
All posts
