Synadia
Welcome back to our next customer story here at RethinkConn. We’re talking today to Christopher Coco from Verrus. Chris, thanks for being here. Can you tell us a little about yourself, your role at Verrus, and what Verrus does at a high level?
Christopher Coco
Thank you for having me — I appreciate being here and I’m looking forward to the conversation. Verrus is a third-party data center company with a different approach to building data centers. Verrus data centers are generally about 30% more efficient, which we achieve by being less resource-intensive than traditional data centers. We harness the power of battery energy storage — we use our battery as a shock absorber so we can be a useful asset on the utility grid. We can flex power demand down when necessary by using the battery, so we can be better participants in programs like demand response, all without touching the customer’s IT load. Ideally the customer helps with this through an interface we expose to them, but they don’t have to participate. It’s an interesting way to make the large load of a data center flexible on the utility grid without having to touch IT load — a novel concept that comes with some design challenges around electrical topology. But it’s a fun challenge and a fun process.
Synadia
Very cool. I imagine life at Verrus is go-go-go right now, as data centers are so central to today’s AI wave and AI economy.
Christopher Coco
You’d be correct — it’s an interesting time. The company is around three years old, and I started when the company started. I don’t come from data centers; I spent the majority of my time before Verrus at a small company called Twitter, working in the platform infrastructure group scaling our RPC stack. When I started at Verrus, the conversation around data centers was, “OK, we’re reaching a power constraint, this has gotten kind of nutty.” In the three years since, it has completely exploded — gone to a place I don’t think we could have imagined — and the power-constraint problem has gotten much worse. The initial problem we were trying to solve has become much more of an issue than we could have imagined.
Synadia
Solving important, high-profile challenges — love it. Before we get into the architecture and how you’re using NATS, can you give us a sense of the shape of the problem when it comes to real-time data flows in a data center context? What do you care about? What’s challenging?
Christopher Coco
There are a few things we’re trying to do. We want to be much more transparent as a third-party data center company than most third-party data centers are. We’d like to give our customers a real-time view of what’s happening with the electrical topology, so they can easily graph their IT load versus power utilization. That’s not something a third-party data center normally makes available, let alone in near real time. So there’s a whole aspect of exfiltrating telemetry data from on-premise out through an interface the customer can access. Then there’s a lot we want to capture around devices and predictive maintenance. We use a battery energy storage system, so understanding how the battery is performing over time — pulling statistics, capturing that, being able to respond to different things. And then there’s a whole messaging backbone we need to bring to bear: from accepting the demand-response utility request, to proxying those to a customer, to doing lots of other things in that plane between our own microservices.
Synadia
Verrus is relatively young — you said only three years old. Was NATS part of your architecture from the outset, or how did you come to it and settle on it? Maybe start by telling us what part of the stack NATS fits in.
Christopher Coco
It was not part of the original stack. It’s funny — I’ve been in technology a very long time, coming from what’s typically referred to in this space as the IT side, not the OT side, which just means more traditional software programming. I have a lot of friends in this area, and friends at other companies were mentioning this pub/sub technology to me. I knew we needed an event-based architecture and we were looking at different things. The standard technology in this space is MQTT, using MQTT brokers, and we were trying to figure out how to make those highly available. That’s not native to the MQTT protocol — clustering those brokers is something you usually have to pay for, and when we looked at vendor solutions, we didn’t want to be locked into a vendor. Open source solutions didn’t cluster. Friends kept talking about this NATS thing, and I started looking into it. The more I looked — I watched all the videos from Jeremy — the more I realized it was more than just pub/sub. It’s hard to understate how revolutionary it seems, even though it’s built on very simple principles. That’s what’s fascinating: the places you can take it and the things you can do from just basic principles is amazing. And the fact that it’s open source — I worked in an open source office at Twitter for many years, I really believe in open source, my team open-sourced all our RPC stack — it’s near and dear to my heart. I love that it’s out there for people to use freely, but there’s also an entity behind the scenes willing to help. We’ve contacted and are working with Synadia to help us out. And getting started was just dead simple.
Synadia
Yeah, we hear that often — NATS is dead simple. You maybe even underestimate at first all it can do, and then it can take you a long way. You talked about knowing you needed an event-based architecture, realizing MQTT wasn’t fit for your purpose, and finding NATS. Can you tell us more about where it fits in — what other pieces of the stack it’s interoperating with, and what job it does for your data flows?
Christopher Coco
In the OT world there’s a concept called the unified namespace. It took me a while to figure out what it meant, but it basically means: all these systems collecting data on a factory or plant floor used to be connected point-to-point. The big change these days is to have a publisher publish into a universal — unified — namespace, and any consumer consume that data if they’re allowed to. NATS fits that amazingly well, whereas MQTT actually doesn’t, because of its statefulness. Even talking to the inventor of MQTT, he mentioned to me that it was really meant for command and control, not this stateless publishing it’s been wedged into. So for us, NATS forms the backbone of our unified namespace. It’s essentially the data plane from each data center into our cloud topologies. We collect data in real time from each data center into our cloud, where we do predictive and optimization analysis, and then shunt some pieces of that data into a customer interface they can subscribe to. It’s a major portion of that backbone. What’s also interesting is that it’s not just streaming telemetry — it forms a data plane where we also have microservices that ask questions. Because it has not just pub/sub but request-reply semantics, which is huge, we can build microservices on the same data plane that query each other or query data. So operationally we have not only a telemetry system but also a queryable data plane that rides the same substrate. It’s not an additional system built alongside the telemetry part — and that’s huge for us.
Synadia
When I’ve read about OT/IT in the past, it always starts with telemetry — getting data off the devices up to the cloud for some next step. And very quickly it becomes: can we go the other way? Can we interrogate the devices, have a two-way street? It sounds like you’ve gotten there, in part with NATS.
Christopher Coco
Yes, and that enables some fascinating things. I won’t get too deep into all of them, but it lets us do this securely and safely, using the zero-trust mechanisms we’re adhering to in our architecture. That’s another thing that’s amazing out of the box — the fact that it comes with nkeys and the security model NATS brings, which follows exactly the things we’re trying to accomplish. It made it very simpatico. It’s weird — I keep getting blown away by the feature set. Every time we delve into another piece, it’s like, “oh, that’s already there.” You start with a small part — we started with the CLI — and now we’re making heavy use of JetStream and the KV store, and we’ve been kicking around ideas around the object store. We’re potentially building an entire digital twin over the NATS substrate. Digital twin is a hot topic and I hesitate to use the word, but it’s easier for people to understand what I mean. The amount of things you can accomplish with such an elegant system is really fascinating, and it has made this job in an OT space that much more exciting.
Synadia
Can you tell us anything about the NATS adoption and scaling milestones at Verrus? What was it like once you found NATS, selected it, and realized it might be the right tool — how did you fit it into your architecture and go from a POC to production?
Christopher Coco
One of the first things we looked at was using it for service discovery between microservices. That was the quick POC. We were looking at MQTT clustering, and this could be a potential change for that, but what was really interesting was not having to define our own service mesh. We have a small set of microservices that need to talk to each other, and we didn’t have to deploy a whole Istio or pull anything off the shelf. We realized we could use subject-based messaging, and suddenly we had this service mesh, getting our services to talk to each other through NATS. Then it ballooned. We started wondering what else we could put through the system. We got into telemetry, but then looked at the OT protocols we have to use inside the data center — again, point-to-point; you have to know the IP address and port, speak Modbus, and so on. These protocols aren’t very “web”: the client and server have to know about each other and hard-code addresses. So we thought, can we build proxies through NATS? Can we proxy these over NATS messaging? It’s been fascinating. Now we can refer to devices by a logical subject name, have NATS map that name to a current IP and port, and if we need to update it, we simply update the mapping in NATS without any customers or clients needing to know. That gives us operational flexibility. It’s one of these things that’s easy to do through NATS — even though it’s kind of complicated — and it gives us crazy powerful semantics. We now have a topology that doesn’t have to be point-to-point, which is what we’re trying to get away from, and it’s not just our microservices — it’s even microservices or other things talking to actual devices.
Synadia
That example — proxying the OT protocol through NATS with a mapping table — is a pattern I hadn’t heard before. It highlights the flexibility NATS brings: simple primitives you can push pretty far. Engineers love metrics — can you tell us anything about benchmarking, scale testing, throughput, or the criteria you looked at when rolling out NATS?
Christopher Coco
It’s interesting because we don’t have a lot of scale, which is very different for me coming from where I came from. The issue for us isn’t necessarily scale — it’s correctness and security. Those are the things we look at when evaluating technology. What still highlights the usefulness of NATS is that we get that correctness and security somewhat out of the box: request-reply semantics, queue groups, interest topics, atomic matching if necessary. NATS has a lot of semantics that let us enforce correctness, and do it securely. While scale isn’t necessarily what we look at, we know it’s scalable — we can cluster across regions, even across cloud vendors if necessary. It’s not something we’re doing right now, but the possibility is interesting. We use leaf-node-to-cloud for clustering, and super clusters across regions. All of that is one system — it doesn’t take four or five different things and a couple of operators to put together. It’s mind-blowing that these simple principles blow up into this large machinery. It started with a simple publish to this subject, subscribe to that subject, and now there’s a massive system behind it.
Synadia
I always return to the analogy of the NATS server as an incredible Lego brick — you can put together all sorts of arbitrary, sometimes seemingly bizarre topologies just by snapping bricks together. It sounds like it’s almost become a game of what fun things you can accomplish.
Christopher Coco
Since we’re new and trying to push NATS to see what’s possible, we’re trying to get a little crazy to see where things fall over — where it becomes “OK, we shouldn’t do that.” We haven’t quite hit that yet, which is interesting.
Synadia
How about the operational side — how has running NATS in production been for your teams? Was it an easy transition? Has it changed how they run in production?
Christopher Coco
I’ll say it’s easier than running Kubernetes. It’s been great. One thing we’re also using from Synadia is the control plane, to help bootstrap our NATS clusters. There’s a fully exposed API we integrate with through our orchestration layer, so we can bootstrap all the credentials and base-level things we want in the cluster. We have that wired up with Crossplane to actuate deploying those NATS clusters. Operationally it’s been — “pleasant” isn’t quite the word, nothing is operationally pleasant — but it’s been better than other things we have to deal with. It’s pretty solid.
Synadia
What’s the story as Verrus grows — adding customers, working at more data centers? How does NATS scale with you when onboarding customers?
Christopher Coco
NATS will scale. We have an on-premise cluster in each building, which leaf-nodes to the cloud topology. As we add buildings, those come online and we’ll likely grow our super cluster. We may even partition into sub-super-clusters — some section of buildings per campus or site talking to a NATS cluster that’s potentially super-clustered, depending on what we need for compute in those layers. We don’t foresee this not scaling. With these complicated Lego blocks, we’re already thinking about how to sub-segment and subdivide into logical groupings that make sense — but it won’t matter to the customers. It’s not like we have to go touch the data center to do so. Operationally it’s somewhat simple to put together. Simple doesn’t always mean easy, but it’s a straightforward thing we think we can do as we grow — and much better than trying to cluster stateful MQTT.
Synadia
We touched on interoperability with the OT side and proxying protocols. Anything else on interoperability — what else does NATS have to play nice with in your stack?
Christopher Coco
We mostly write our application stack in Rust, and we appreciate that even though NATS itself is Go, there are SDKs for those of us writing in Rust. We also integrate with our SCADA system, which is in Java, and we appreciate that there’s a NATS SDK for Java so we can plug a module into that SCADA system and go directly into NATS. Interoperability from a language perspective — I’m a software engineer, so I’ll talk about languages — we like to be polyglot, and NATS really supports that out of the box. As for other things in our stack, it’s a simple protocol — headers and bytes at the base of the semantics — which is wildly useful. Going back to your Lego block: that simple brick means you can construct so many different things and interact with so many different things. If it’s on the network, we can shove it through NATS, because we can always put bytes through NATS. It’s been a fascinating “what can we do with this until we break it” journey. There are probably some things we’ll get to that we realize we shouldn’t do, but we haven’t quite gotten there yet.
Synadia
I think I have two closing questions. First, anything on your wish list or forward-looking — if you could wave a magic wand and make NATS do X, where are you looking next?
Christopher Coco
It’s an interesting question. Every time I think we need something, I look and realize it’s already there. Our ideas aren’t new — someone has thought of it before and decided it needs to go into NATS. One thing we want to ensure is that all our data centers are shielded from each other — no east-west communication across them. They should connect to the hub and talk through the cloud, but we don’t want to accidentally enable them talking to each other through the cloud. That’s a solved problem — there’s now an actual primitive and configuration you can set up to make sure you do that, to shunt off that east-west communication. That’s fascinating. I don’t think we’ll ever get down to running NATS embedded on a device, but I think that’s a fascinating place NATS might be useful. Could NATS be a different D-Bus? I don’t know, but it’s quite possible. And it’s fascinating what that means from the Lego-brick analogy — if it’s running at that layer, and it’s all NATS above it too. I don’t have anything specific right now, because we’re still neophytes pushing the envelope. As we mature, we’ll find things where we think “it’d be great if this existed” — but by the time we get to it, it’s probably already done.
Synadia
It feels like we’re catching you at a good moment in your NATS journey. Running NATS at this IT/OT convergence is becoming more common. Do you have any takeaways or learnings — foot-guns to avoid, or things you learned over the last couple of years — you’d want to share with the audience?
Christopher Coco
As much as I’m touting NATS, the lesson is to make sure you’re using the right tool for the right job. We’re still figuring out exactly where NATS fits, where MQTT fits, where different things fit — finding the right protocol for the right layer. We think MQTT is still very useful at the edge; it’s an OT standard, a lot of devices speak MQTT, and we can’t and don’t necessarily want to get away from that. So we’ll likely still run MQTT at the edge. The fact that NATS supports MQTT and even Sparkplug B is very useful — someone beat us to suggesting that — so we can run NATS effectively as an MQTT broker. My advice is: everything’s a nail, and I don’t want NATS to be our hammer. The fact that we’re hammering a lot of things with NATS right now and it seems to be working is fascinating and extremely useful. But always second-guess yourself, and make sure you’re doing something for the right reasons — not just because it’s there and you want everything to work one way. We try to be rigorous about how we use things. NATS is a very useful tool — it’s very cool and very interesting — but it probably won’t be the right thing for every spot everywhere.
Synadia
Yeah, it does have its limits. [laughs] Don’t let anybody hear you say that. Chris, thank you so much for taking time to chat and tell us about your NATS adoption journey and working with Synadia. You laid out some really compelling arguments, and it’s exciting to see NATS used in this data center space that’s so central to our tech world right now.
Christopher Coco
Thank you for having me. I appreciate the time.
