Not so fast: Rate-limiting a runaway NATS publisher with Synadia Protect

Imagine you’re running the backend for a fleet of machines and services spanning edge sites and your cloud environments. Each site connects to your hub as a leaf node, and publishes telemetry like temperatures, counters, and run-states resulting in thousands of messages per minute from each site.

At 2:00am, one site completes a firmware update but something in the new collector’s buffering logic goes wrong and it starts republishing its entire spool in a loop. The site is now pushing 200 million messages per minute into a cluster that the whole enterprise depends on.

The real bummer is that the leaf node was doing exactly what it was permitted to do, and took down the entire backend in the process. Nothing in its path was able to prevent the outage. That’s why we’ve added rate-limiting to Synadia Protect to prevent unexpected client or leaf node behavior from overwhelming your central NATS hub.

Where does runaway traffic come from?

A firmware bug is one cause. Similar traffic spikes happen when:

• Crash-looping pods replay their checkpoints on every restart.
• Publishers retry acknowledgment timeouts without backoff or a stable Nats-Msg-Id, adding duplicates to a stream.
• Clients reconnect together and flush their buffered publishes at once.
• A bulk import sends large messages faster than the deployment can handle, even though each message fits under the payload cap.

Your NATS server has payload caps, account limits, and stream limits. But a connection can stay within those limits while still sending enough traffic to disrupt other workloads. Operators also need a way to bound how quickly each connection can push data.

Rate limiting in Synadia Protect

With the release of version 1.0.1, rate limiting is included in Synadia Protect. It caps how much traffic any single connection can push into a cluster, or pull out of it by message count or payload bytes. Every connection gets its own allowance, and when a connection exceeds its allowance, the gateway disconnects or suspends it.

Protect rate-limiting ships as a set of built-in rules that are activated in a policy bundle like any other, and like the rest of Protect, it works in front of any NATS deployment.

Rate is just another fact

Protect already parses every CONNECT, PUB, MSG, HPUB, HMSG and LMSG that passes through it, evaluating each one against the active bundle, and already keeps state for the life of a connection.

Under the hood every limit is a token bucket held on the connection handler rather than in any shared structure. Four per connection: messages and bytes, in each direction.

Two config values, no expressions to write

You don’t need to write an expression to use any of this. The built-in rules are named for exactly what they do, and each one takes an interval and a cap:

1
activations:
2
  com.synadia.protect.builtins.v1.deny.ratelimit.connection.ingress.messages.message: true
3
  com.synadia.protect.builtins.v1.suspend.ratelimit.connection.egress.bytes.message: true
4

5
configurations:
6
  com.synadia.protect.builtins.v1.deny.ratelimit.connection.ingress.messages.message:
7
    interval: 1s
8
    messages: 5000
9
  com.synadia.protect.builtins.v1.suspend.ratelimit.connection.egress.bytes.message:
10
    interval: 10s
11
    bytes: 50mb

Ingress counts what the client sends toward the backend; egress counts what the backend delivers to it. messages count protocol messages; bytes count payload bytes. deny closes the connection; suspend keeps it open, but stops forwarding its traffic for a set period of time. Every combination applies to both client and leaf connections.

Intervals are durations (500ms, 1s, 5m, etc) and a bare number is read as seconds. Byte caps take 100kb, 1mb, or a plain number of bytes. Both values are required and both must be greater than zero. The bundle is validated at install, so a bad value is rejected before it ever reaches a gateway.

Auto-refill limits at a known rate

A limit of 5,000 messages per second gives the connection a balance of 5,000 credits, refilling at 5,000 per second. Each message spends one credit. A byte limit spends one credit per payload byte.

A connection that’s been quiet can spend its whole allowance at once and then keep going at the steady rate. A connection that runs above the steady rate for long enough drains the bucket and trips the rule. What’s enforced is the sustained rate, not the spacing between messages. So a site that batches its telemetry into a burst every 30 seconds doesn’t get punished for the burst.

Deny hangs up. Suspend puts a connection on hold

deny closes the connection with the rule’s message attached. The client sees a clean error stamped with a correlation ID (traceable in the logs for support and debugging), then reconnects on its own and comes back with a fresh allowance. This is the right default because the client that tripped the rule finds out why it was cut off, and it finds out in one place with the rule’s name on it. One thing to expect from chatty clients is that many NATS clients buffer publishes while disconnected and flush them the moment they’re back online, so a denied connection can return with a burst of activity. The fresh allowance absorbs one interval of it and the rule trips again on the rest, which is the correct outcome but looks like flapping in the audit log.

suspend leaves the connection up and stops forwarding its traffic. From the client’s side it looks like a healthy but silent connection — it still gets PING responses, so it believes it’s connected, while requests time out and subscriptions go quiet. It stays parked for the port’s suspend timeout, 5 minutes by default (default_suspend_timeout), then gets closed. Rule evaluation stops for a suspended connection, but a trace profile still captures every byte it tries to send. Reach for suspend when you want a client held off the backend for a while rather than reconnecting immediately, when you want to watch what it does while it’s held — and when the client’s reconnect behavior is itself part of the problem. A fleet of crash-looping publishers without reconnect jitter will synchronize into a thundering herd against your gateway; suspend gives them nothing to synchronize on.

See it in action: one runaway leaf node contained

Same 2:00am firmware update from the top of the post, with an ingress message cap active on the leaf port.

The limit trips. The collector’s republish loop drains its 5,000-credit bucket in the first second. Message 5,001 is denied and the leaf connection closes with ingress message rate exceeded. The backend never sees the flood. The other sites keep publishing normally, because the allowance is per connection and theirs are untouched.

The SIEM knows before you do. A policy.action event lands with action deny, the offending connection ID, the rule reference and the bundle version, correlated by the same cuuid the gateway handed the client in its error response. The page/alert that would have woken you at 2:00am is now a line in a dashboard at 9:00am that tells you what’s wrong.

You look at what it was doing. Switch the port’s activation from deny to suspend and the next offender is held instead of hung up on. Attach a trace profile to the suspended connection and you capture every message it’s still trying to publish, at the protocol level, while none of them reach the backend. That’s the evidence you send the vendor.

Tune the limit. Adjust the initial 5,000-message limit based on observed traffic and activate a new bundle version without restarting the gateway. New connections receive the updated limit; existing connections must reconnect to pick it up.

The evidence keeps arriving. Every denial since the rule went live is in the SIEM, tagged with the exact rule, bundle name and bundle version. When an auditor asks you to show what your control caught and when, the answer comes from the same stream that powers the operator’s dashboard.

Before you turn it on

One action per direction and unit

Activating both the deny and the suspend variant for, say, ingress messages does not give you a warning followed by a hangup. Both rules share the same bucket and each one spends a credit, so every message is charged twice and the effective limit is half what you configured. Pick one action per direction and unit.

Egress limits bind the receiver, not the sender

An egress cap acts on the client that is receiving. A subscriber on a busy subject can trip an egress limit because of what the hub is sending it, not because of anything it did. That’s the intended use — bounding what a leaf or client is allowed to consume — but it’s the opposite of the intuition that a rate limit punishes the noisy party.

Running connections keep the allowance they started with

A limiter is parameterized once, when it’s created, and it lives on the connection handler across bundle reloads. Installing a bundle with different numbers changes what new connections get; connections that already have a limiter keep their original values until they close. If a change needs to take effect everywhere now, cycle the clients.

A few smaller things worth knowing:

• bytes means payload bytes. Subjects, headers and protocol framing aren’t counted.
• A single message larger than the byte cap always trips the rule on its own. Set byte caps above the largest message you expect.
• Limits are per connection, not per account or subject. Which means that a hundred leaf nodes could get a hundred separate allowances.

Get started

Rate limiting is now available in Synadia Protect 1.0.1. If you’re not already running Protect, start with a 30-day free trial — install a gateway, write a bundle, and enforce your first rate limit against a live NATS cluster.

• Protect Trial
• Protect Docs
• Introducing Synadia Protect