From Device Telemetry to Autonomous Agents: 200 Million Events per Minute on NATS

Calix is an AI platform company that helps broadband and internet service providers transform their operations and accelerate delivery with a differentiated subscriber experience. Its focus is increasingly agentic: enabling autonomous workflows that make provider operations more versatile and hands-off.

Calix also designs and supplies the hardware at the edge of those networks — residential gateways (the routers in subscriber homes), optical network terminals, and other access devices. Managing the operations of that fleet efficiently from the cloud means ingesting an enormous, continuous stream of events from devices in the field.

Device communication at Calix runs in two directions: telemetry flowing up from devices (speed, quality of experience, health) and configuration flowing down from the cloud. The previous system moved this over WebSockets between cloud and device. It worked — but it didn’t scale the way the team needed, and it introduced friction that grew with the fleet.

• Scaling ceilings — the WebSocket architecture couldn’t grow smoothly with a fleet climbing toward millions of devices.
• Slow device onboarding — bringing new devices online took significant time and was a recurring pain point.
• Operational complexity — when a connection dropped, recovering and maintaining operations was a nightmare, and the infra team lost significant time to firefighting.
• Real-time latency requirements — events happening on a device at 12:00 needed to be in the cloud at 12:00:01, without exception.

Rajaram, who introduced NATS to Calix, first considered MQTT given his prior experience with it. But MQTT carried its own baggage: fragmented versions, no strong open-source option that avoided leaning on a vendor, and a range of interoperability work required to make it viable at Calix’s scale.

Studying how system-to-system communication should work, NATS stood out. He ran a proof of concept entirely on his own laptop — spinning up a leaf cluster, a core hub, and a simulated home device in a local Kubernetes environment — and proved to the company that a real solution existed before a line of production code was written.

NATS runs the same on a laptop as it does at the edge or in a massive cloud cluster — and that was exactly Calix’s experience moving from PoC to production.

Calix didn’t adopt a single off-the-shelf topology. The team engineered a “bundle” concept — a repeatable unit combining core and leaf nodes with mirroring and sourcing — that scales elastically as devices come online. Onboarding a million more devices means adding more bundles, not re-architecting. Helm charts and a custom wrapper let the infra team scale bundles up cleanly; because event volume only ever grows, there’s no scale-down to manage.

On top of that substrate, Calix standardized on three NATS communication patterns it calls its reference patterns — the mantra the whole architecture returns to:

• Request–Reply (synchronous) — the cloud requests, a NATS client answers, and a response comes back. Used heavily on the configuration path.
• Key/Value (state) — keys pushed at will when the cloud makes a change, for example telling a device to report data on a new interval.
• Pub/Sub (async) — highly configurable, per-device subjects that scale elastically for high-volume telemetry.

JetStream as the state layer

For Rajaram, JetStream was the game-changer. NATS wasn’t just a communication engine — it gave Calix a durable state layer, letting the team process data on its own terms: some streams handled immediately, others deferred a few minutes by priority. JetStream became the glue between the up (telemetry) and down (configuration) paths, and the team made it scalable across all parts of the topology.

Auth callout to keep bad actors out

A key concern was preventing a bad or compromised device from reaching the cloud. Beyond authentication, Calix needed authorization at the door. With Synadia’s help, the team built a scalable auth callout service — callout pods that filter connections on device attributes before anything enters the ecosystem, scaling seamlessly alongside the rest of the NATS deployment.

C++ on constrained devices, Java in the cloud

Resource constraints drove client choices at the edge. Residential gateways are highly resource-sensitive, so the NATS client there is written in C++; larger optical network terminals have more headroom. On the cloud side, Calix stayed with Java so engineers who built the previous system could migrate seamlessly. The original PoC was written in Go — often the first stop for NATS users — before the team moved to C++ on the device.

NATS gave Calix a single ecosystem — core, JetStream, KV, auth callout, and leaf nodes — to replace a brittle WebSocket fleet and consolidate device communication onto one data plane. Instead of reaching for many separate tools, the team could tune NATS again and again.

• Predictable, repeatable onboarding — the bundle model makes onboarding a million devices a known scaling operation.
• Real-time at scale — ~200M events/minute reach the cloud in real time, with latency measured in a single second end to end.
• Near-zero infrastructure issues — since going to production on NATS, the team reports essentially no infrastructure problems, versus constant firefighting before.
• Drastically fewer microservices — the number of pods and services on the configuration path dropped sharply while the reference pattern stayed the same.
• Out-of-the-box capabilities — rate limiting and observability came built in, avoiding extra load balancers and bolt-on tooling.
• Fast time to production — despite tight deadlines, NATS was never the blocker; expert support compressed a multi-year learning curve.

Getting device data into the cloud in real time is the precondition for everything Calix is building next. As an AI platform company, the payoff of the NATS migration is the ability to build autonomous agents on top of a reliable real-time data plane — something that was relatively tough before.

A simple example: a speed-test agent that talks directly to a device through NATS, rather than waiting on slow legacy paths. More ambitious work involves running agents against in-flight data. It all points toward a broader vision the team calls Network Data as a Service (NDaaS) — making any available network data queryable and serveable over NATS, feeding Calix’s north-star goal of agents running autonomously.

Looking ahead

The team is exploring leaf nodes on the devices themselves — pushing collection and transformation to the edge so the cloud can relax, with offline capability as a bonus. It’s also working with Synadia on gNMI integration with NATS, aligning the deployment with where the telecom industry is heading for device-to-cloud management.

NATS is open source, and Calix could have learned it alone — but Rajaram estimates that path would have taken two to three years. The team chose Synadia for the expertise. When a bug appeared, it was fixed in days, not weeks. When a hard problem needed debugging, Synadia engineers joined a working session — once at 2:00 a.m. Calix’s infrastructure lead pointed to the depth of documentation, books, and the NATS Slack community as making onboarding dramatically easier. Just as valuable was the ability to mock architectures at small scale in Docker before scaling them up for real.

The team’s advice to others building large distributed fleets is consistent: understand the fundamentals first. Learn how mirroring and sourcing work, and which NATS components fit your use case — you may not even need a leaf. NATS gives you a large toolbox; the skill is choosing the right pieces for the job.