Preventing Duplicate Publishes While a Matching Message Exists in JetStream

A common community question is how to publish a recurring message, such as one sent every five minutes, while preventing a second copy from being accepted if the first copy is still present in a JetStream stream.

For this use case, Nats-Msg-Id may not be the right fit. It is useful for duplicate publish suppression within a configured duplicate window, especially for retry safety, but it is window-based. If you want the stream to reject a publish based on whether a message for the same logical item currently exists, use a JetStream publish expectation instead.

Use the subject as the deduplication identity

JetStream supports conditional publishes using expectation headers. For a create-if-absent pattern, publish with:

1
Nats-Expected-Last-Subject-Sequence: 0

This tells JetStream: accept this publish only if the last sequence for this exact subject is 0, meaning there is no existing message for that subject in the stream.

That means the subject itself becomes the deduplication key. There is not a separate message ID for this pattern. You encode the logical identity you want to protect into the subject.

For example, instead of publishing all work to one subject:

1
tasks.reconcile

publish each deduplicated item to a stable, item-specific subject:

1
tasks.reconcile.customer-123

Then publish with the expectation header:

1
subject: tasks.reconcile.customer-123
2
header:  Nats-Expected-Last-Subject-Sequence: 0
3
body:    { ... }

If a message for tasks.reconcile.customer-123 already exists in the stream, JetStream rejects the publish and the duplicate is not accepted. If no message for that subject exists, the publish succeeds.

The rejection is reported in the publish acknowledgment, so use a JetStream publish call that waits for and inspects the PubAck. A core NATS fire-and-forget publish does not surface the result, so the application would not learn that the message was refused. On rejection the publisher receives a publish error (a wrong last sequence error); treat that as “a message for this identity already exists” and handle it deliberately rather than retrying blindly.

This guarantee holds only if every publisher to that subject sets the expectation header. A single publish that omits Nats-Expected-Last-Subject-Sequence is accepted unconditionally and can still create a duplicate.

Consumers can still subscribe using a wildcard, for example:

1
tasks.reconcile.*

or another subject pattern that matches the subjects stored in the stream.

How this differs from Nats-Msg-Id

Nats-Msg-Id and Nats-Expected-Last-Subject-Sequence solve different problems.

Use Nats-Msg-Id when you want idempotent publish retries within a time window. Use the subject sequence expectation when the acceptance condition is based on whether a message for that subject already exists in the stream.

Alternative: enforce one message per subject at the stream level

The expectation header is a per-publish, opt-in check. If you would rather have the server enforce “at most one message per subject” for every publisher automatically, you can configure the stream itself instead of relying on a header.

Configure the stream with all three of these settings:

• discard: new
• max_msgs_per_subject: 1
• discard_new_per_subject: true

With this combination, once a subject already holds its one allowed message, JetStream rejects further publishes to that same subject instead of discarding the older message. Without discard_new_per_subject, a per-subject limit discards the oldest message for the subject, which replaces the existing message rather than rejecting the new one. You still encode the logical identity in the subject, and a rejected publisher still receives a publish error in its acknowledgment.

Tradeoffs:

• The expectation header works on any stream and is decided per publish, so individual publishes can opt in or out, and the same header can carry non-zero values for optimistic-concurrency updates. It depends on every publisher setting the header consistently.
• The stream-configuration approach applies uniformly to all publishers with no header required, but the one-per-subject limit then applies to every subject in the stream, and discard_new_per_subject requires NATS server 2.10 or newer.

Both approaches still depend on the stream’s retention behavior to define when an existing message is considered gone, which is covered next.

Retention policy matters

The phrase “still exists” is defined by the stream, not by the publisher.

A message may remain in a stream after delivery depending on the stream retention policy and limits. For example:

• With limits-based retention, where a message is not removed simply because it was acknowledged, the conditional publish may continue to fail until the message ages out, is removed by stream limits, or is explicitly deleted or purged.
• With work-queue style retention, acknowledged messages are removed from the stream, so a later publish for the same subject can be accepted after the prior message is processed and acknowledged.

Choose the stream retention behavior that matches the lifecycle you want. If your rule is “only one outstanding work item per customer,” then the stream should remove the old item when it is no longer outstanding, or you should explicitly remove it as part of your workflow.

Avoid client-side check-then-publish races

A common mistake is to query the stream first and then publish only if no matching message is found. That creates a race: two publishers can both check, both see nothing, and both publish.

The expectation header makes the check part of the publish operation handled by JetStream. Concurrent publishers using the same subject and Nats-Expected-Last-Subject-Sequence: 0 will not both create the first message for that subject.

Practical design guidance

When using this pattern:

1. Pick a stable subject token for the logical identity you want to deduplicate, such as a job ID, tenant ID, customer ID, or task key.
2. Do not include changing values, such as timestamps or random IDs, in the deduplication part of the subject.
3. Configure the stream to capture those subjects, for example tasks.reconcile.* or tasks.reconcile.>.
4. Enforce the rule with Nats-Expected-Last-Subject-Sequence: 0 on every publisher, or with the stream-level discard_new_per_subject configuration, and check the publish acknowledgment so rejected publishes are handled.
5. Make sure retention, acknowledgments, deletes, or expiry remove the existing message when you want a future publish for the same identity to be accepted.

Summary

If you need time-window deduplication, use Nats-Msg-Id. If you need JetStream to reject a publish while a message for the same logical item still exists, encode that item identity in the subject and either publish with Nats-Expected-Last-Subject-Sequence: 0 or configure the stream to enforce one message per subject with discard_new_per_subject.

Either way you get a server-side create-if-absent pattern without a client-side race, as long as the subject design and stream retention policy match the lifecycle you want and every publisher participates consistently.

Want help from the NATS experts? Meet with our architects to get help tailored to your use case and environment.