NATS Server 2.15: How we scale and move streams safely

We’ve recently released NATS Server 2.15, which focused entirely on reliability and improving operations, including fully reworking how scales, moves, and peer-removes work, as well as improving stream backups, stream source recreate detection, and higher performance when using sync: always for replicated streams. But today, I’d like to give you some insights into what it took to fully rework the server’s meta layer, including all the nasty edge cases that needed solving, and how NATS Server 2.15 fixes them.

Context

A NATS system consists of three “layers”: the meta layer, streams and consumers. Each layer functions independently under its own peer set, which allows streams and consumers to continue to function even if there is no quorum on the meta layer (i.e. when no new stream or consumer creates, updates or deletes can be performed).

The meta layer has previously been, and still is, the place where decisions are made about which servers any given stream and consumer are hosted on. However, previously the responsibilities of managing the stream and consumer peers (which servers host the asset) were shared between the meta layer and the stream/consumer layer. When scaling up, scaling down or moving an asset, the interaction between these layers could result in temporary disagreements about the peer set in the best case, and potentially losing data in the worst case, typically when the system is already running in a degraded state.

Scaling up/down and moving streams and consumers can’t be that difficult, right?

In their simplest form, scaling up/down and moving streams and consumers means: new peers need to be added, data needs to be replicated, old peers need to be removed. Although that may sound simple enough, it’s deceptively tricky to implement correctly as there are many scenarios to consider that may not be immediately obvious.

Scaling up

Scaling up is quite straightforward: you add new peers and catch them up. However, even here it’s important to do the peer changes correctly. Raft requires that only one membership change happens at any given time, so multiple membership changes need to be handled sequentially. Taking a shortcut by switching the peer set in one go could otherwise result in split-brain, where one part of the cluster could accept writes that the other doesn’t have. The previous design, with responsibilities shared between the meta layer and stream/consumer layer, took exactly this shortcut; the meta layer forced immediate peer set changes, while the stream/consumer layer still did peer management independently as well.

Scaling down

Scaling down requires you to select which peers to scale down to. This can’t just be any peers, because you need to prioritize peers that have data over ones that don’t, otherwise you could lose some or all data. This was previously implemented by the meta leader requesting, over the network, which server was currently the stream/consumer leader. But, as far as the meta leader is concerned, the stream/consumer could be “on the other side of the world”. Would a delay in the request/response change anything? What if the stream leader changes and accepts new writes right after the previous stream leader had already responded to the meta leader? What if the request would time out and the meta leader never received a response? What if there was no stream leader at all?

Perhaps this helps to illustrate that any design where anything other than the stream/consumer leader itself decides the peer set will be fundamentally flawed. It’s not surprising, then, that in the worst case where the meta leader got no response, it would have to “guess” which peers to keep, which would essentially be a coin-flip whether the data would be preserved. However, you might say: if the meta leader doesn’t get a response, then it could simply reject the scale request? While that indeed sounds more correct, it would require the user to retry their request for what is likely a temporary condition. And, it still wouldn’t resolve the case of a stale leader responding and ending up with the wrong peer set being chosen anyway. A fundamental design flaw can’t be fixed with simple workarounds: the design itself needs to change.

Moving consumers

To make things more difficult: scaling down a stream to a single replica also means that consumers need to be scaled down. You’ll first need to scale down the consumers before you can scale down the stream, and this needs to be done in the right order to avoid data loss. But, it would be too easy if the replica count of consumers always matched that of the stream, right? So, let’s up the ante by having a 3-replica stream with a 1-replica consumer, and specifically where the stream and consumer leader aren’t on the same server. This is the hardest edge case for scaling down a stream, since such consumers first need to scale up to the stream peer that we intend to scale down to, then scale the consumer back down, and only after all that can the stream finally be scaled down.

This is another area where the previous design allowed the meta leader to take shortcuts. Instead of scaling up the consumer first and ensuring all is replicated properly before scaling down the consumer and stream, the meta leader would force all of that to happen in a single shot. The meta leader would ask which stream leader to preserve, and then iterate through all consumers that needed to “move”, i.e. the single-replica consumers that would be removed by the stream scale down. For each consumer, the meta leader would request the consumer state and encode that in the new consumer assignment. However, again, this has a fundamental flaw in that consumers can still be used while this is happening, so the captured state would often be stale and partially reverted. And again, what happens if there is no response to this request? For example, if the node that hosts the single-replica consumer is offline? In that case, the consumer would still be moved to the single-replica stream node we intended to scale down to, but all its state would be lost. Gosh, these shortcuts can be painful.

Moving streams

Finally, streams can be moved between clusters. For example, moving a stream and its consumers from cluster A to cluster B. I won’t spend too much time on this, as it inherits many of the same issues mentioned above. But it’s important to only move consumers once the other cluster hosts the stream and to not remove streams from the old cluster until all consumers have moved off.

Ideally, consumers should not only be moved once the other cluster has been assigned the stream but also after it has already caught up to ensure that the consumers don’t stall after they’re moved. Additionally, new consumers created during this process should be assigned to the old cluster while it has a majority of the data, and only to the new cluster once a majority of new nodes has caught up on the data, ensuring that new consumers don’t stall the overall move. Under the previous design, neither happened.

Phew, that’s a lot!

It’s worth saying that this wouldn’t happen for every scale or move. Normally servers are online and caught up with minimal lag, and meta, stream, and consumer leadership is quite stable. However, the chance of it happening at all was not acceptable to us, so we have prioritized fixing it for NATS Server 2.15.

For quite some time it has been obvious to us, the NATS Server team, that this was a fundamental problem that needed to be addressed. At the same time, this was no simple feat that we could just “patch” in and treat lightly. Much of the difficulty comes from it being a replicated, distributed system. You could compare it to an intricate network of connected strings, where if you undo only one, the whole network comes apart. This needed to be fixed properly by taking everything into consideration, and importantly: no shortcuts allowed!

Put simply:

• The meta layer must always (only) suggest membership changes, and never take unsafe shortcuts.
• The stream and consumer layers are solely responsible for the Raft group’s membership, and safely drive the meta layer’s suggestions to completion.

So, is it fixed?

Yes!

It’s anticlimactic to plainly claim “it works” and call it a day, so I’ll spend the rest of this post explaining how we fixed it and what it took to test that it actually works.

Desired state reconciliation

Since the previous design had serious flaws, the first step is to pick a design that works correctly, and desired state reconciliation is exactly that.

Desired state reconciliation is the process of continuously comparing the actual state of a system with its declared desired state, and taking the actions needed to converge toward it.

This is exactly what the meta, stream, and consumer layers needed to do. The meta layer could still assign stream and consumer peers for initial creation, and allow deletion without requiring additional coordination with other nodes, but live membership changes require the meta layer to suggest them instead of forcing them through. Whereas previously, in the worst case, given offline servers or no quorum, data could be lost or reverted to stale state, the new design degrades safely: at worst, converging to the desired state “takes a bit longer” but the stream/consumer remains available throughout, and data is never lost unless explicitly performed by the user through a stream/consumer delete or peer-removal.

Let’s gradually walk through the same sections as before and highlight how it works, starting from NATS Server 2.15 using desired state reconciliation.

Scaling up

When scaling up a stream or consumer, the meta leader still assigns which peers can be added. This is still required since the meta leader needs to enforce placement restrictions, for example. However, instead of forcing an immediate membership change, the meta leader registers the desired peers. Below is an example of the CLI output for this change:

Cluster Information:

                         Name: ngsprod-aws-euwest1
                Cluster Group: S-R3F-jLyHTNIY
                       Leader: aws-euwest1-natscj1-4 (683ms)
                      Replica: aws-euwest1-natscj1-2, outdated, not seen, (pending)
                      Replica: aws-euwest1-natscj1-3, outdated, not seen, (pending)

   Cluster Migration Status:

                          Status: waiting for catchup to add peer
                            Type: catchup
                         Created: 2026-11-09 16:32:42 (850ms)

                        Replicas: 1 -> 3
                   Desired Peers: aws-euwest1-natscj1-2
                                  aws-euwest1-natscj1-3
                                  aws-euwest1-natscj1-4

If you’ve used the CLI before, you might already have noticed the new Cluster Migration Status section! This describes exactly what is happening in the cluster:

• We’re scaling up from a single replica to three replicas.
• Two new replicas have already been added to the stream assignment, but they are still considered pending until they’re caught up and added to the Raft group. This is what ensures the stream remains available and consistent during the scaling process. Pending peers aren’t required for quorum, so these are asynchronously catching up while the single-replica Raft group is already replicating new writes.
• The stream leader is managing all membership changes, so any peer addition is handled safely step-by-step.
• Once all replicas have been added and no changes are left, the desired state reconciliation is done and the assignment is finalized in its desired state.

Cluster Information:

                         Name: ngsprod-aws-euwest1
                Cluster Group: S-R3F-jLyHTNIY
                       Leader: aws-euwest1-natscj1-4 (6m55s)
                      Replica: aws-euwest1-natscj1-2, current, seen 933ms ago
                      Replica: aws-euwest1-natscj1-3, current, seen 933ms ago

Scaling down

When scaling up, the meta leader can decide which peers to add, but for scaling down it’s impossible for the meta leader to know which peers can safely be removed. When scaling the stream down from three to a single replica, you might notice the absence of something:

   Cluster Migration Status:

                         Created: 2026-11-09 16:45:40 (121ms)
                        Replicas: 3 -> 1

There are no known Desired Peers! The meta leader registers the scale down to a single replica as desired state, but it doesn’t know which peers to select yet. However, the meta leader does know that it’s about a scale down, and which peers were registered in the assignment previously. The stream leader evaluates this, knows it’s about a scale down, and selects from the allowed peer set which peers can be kept.

This is not only safe, but also more efficient. The stream leader not only knows which peers are online, but also which are up-to-date with the latest state of the Raft log. This allows the stream leader to accurately pick only the best peers to keep; the peers that get removed are naturally those that are offline or lagging behind.

Moving consumers

Consumers similarly use desired state reconciliation, and as you might expect: streams are aware of which consumers need to move and only make a change once it’s safe to do so.

Let’s cut to the chase: what happens for a stream with three replicas that is scaled down to a single replica, with a single-replica consumer on a node that is offline (the case that previously lost the consumer state)?

Cluster Information:

                         Name: ngsprod-aws-euwest1
                Cluster Group: S-R3F-0LIzXCpp
                       Leader: aws-euwest1-natscj1-4 (10m17s)
                      Replica: aws-euwest1-natscj1-2, current, seen 217ms ago
                      Replica: aws-euwest1-natscj1-3, outdated, OFFLINE, seen 11.22s ago

   Cluster Migration Status:

                          Status: waiting for consumer 'CONSUMER' to migrate
                            Type: blocked
                         Created: 2026-11-09 22:19:49 (8.60s)

                        Replicas: 3 -> 1
                   Desired Peers: aws-euwest1-natscj1-4

As you might have expected, or hoped: the stream will not be scaled down until after the consumer has moved. However, since the server of that consumer is offline, this will stall until the server is started back up, which ensures the consumer state is preserved. You could either wait for the server to come back and the consumer to be moved automatically, or delete and recreate the consumer, at the cost of losing its state.

Moving streams

Finally, let’s move a stream between clusters, specifically from EU West to US East. First, the placement of the stream is updated to target the US East cluster, which shows up in Desired Peers and as the Cluster change.

Cluster Information:

                         Name: ngsprod-aws-euwest1
                Cluster Group: S-R3F-fd3j79r6
                       Leader: aws-euwest1-natscj1-2 (12.51s)
                      Replica: aws-euwest1-natscj1-1, current, seen 598ms ago
                      Replica: aws-euwest1-natscj1-3, current, seen 598ms ago

   Cluster Migration Status:

                         Created: 2026-11-09 15:00:24 (123ms)

                         Cluster: ngsprod-aws-euwest1 -> ngsprod-aws-useast2
                   Desired Peers: aws-useast2-natscj1-1
                                  aws-useast2-natscj1-2
                                  aws-useast2-natscj1-3

The Raft group then scales up to six replicas, encompassing the nodes in both clusters.

Cluster Information:

                         Name: ngsprod-aws-euwest1
                Cluster Group: S-R3F-fd3j79r6
                       Leader: aws-euwest1-natscj1-2 (7.65s)
                      Replica: aws-euwest1-natscj1-1, current, seen 48ms ago
                      Replica: aws-euwest1-natscj1-3, current, seen 49ms ago
                      Replica: aws-useast2-natscj1-1, current, seen 457ms ago
                      Replica: aws-useast2-natscj1-2, current, seen 468ms ago
                      Replica: aws-useast2-natscj1-3, current, seen 440ms ago

   Cluster Migration Status:

                          Status: adding peer aws-useast2-natscj1-3
                            Type: membership
                         Created: 2026-11-09 15:00:24 (1.26s)

                         Cluster: ngsprod-aws-euwest1 -> ngsprod-aws-useast2
                   Desired Peers: aws-useast2-natscj1-1
                                  aws-useast2-natscj1-2
                                  aws-useast2-natscj1-3

Lastly, the stream scales back to three replicas on our desired cluster.

While all this is happening, the system makes sure to move all consumers of this stream to the other cluster as well. Any consumers that are being created while the stream is being moved will also first target the original cluster and be moved shortly after. However, any consumers created while the stream has mostly been migrated to the other cluster already will target the new cluster and will therefore not need to be moved afterward. This ensures both moving pre-existing consumers and placing new consumers are as efficient as possible.

Wow, that’s impressive!

Indeed, it is! It’s really cool to be able to do these kinds of operations and see exactly the steps that the servers are performing to (safely) move from the current state to our desired state.

Starting from NATS Server 2.15, it’s safe to scale and move streams and consumers as you see fit. It’s guaranteed that no data can be lost (unless you explicitly delete a stream/consumer or in specific cases when using peer-remove), streams and consumers stay available with no downtime when scaling up or down, and there will only be a single stream/consumer leader transfer to the other cluster when moving a stream.

It’s easy to claim “it works” though, so the remainder of this post is about testing.

Testing

The simplest way to start testing this is to add (more) unit/integration tests, which exist in the server repository and run as part of the regular test suite. For all the operations that exist, we can add server tests that exercise the respective code paths (scale up, down, move, etc.), and we verify that all nodes end up in a properly replicated and synchronized state. For good measure we also include some “obvious” edge cases, for example: certain servers being or going offline during the operation, stream catchup temporarily lagging or needing to be retried, the operation still completing in a timely fashion as long as a majority of servers is still online, etc.

While such tests are very useful, and I’d even go as far as saying they are required as a bare minimum, they “only” test for potential future regressions as well as “just” the cases that we know to test for. Generally such test cases are cheap to run, which is why they’re in the regular test suite, but they are nowhere near exhaustive enough to test all possible scenarios. The really “exciting” bugs usually hide beyond the happy path or some “simple” edge cases that are easy to check for. And, since NATS is a distributed system, these may only surface when introducing coordinated failures: server crashes/hard kills, network partitions, both symmetric and asymmetric network latencies between servers and clients, temporarily clogged network links, process/thread pauses, etc.

We’ll need two things for this kind of testing: a workload that exercises the code paths we want to be tested, and an environment that can subject it to all kinds of faults to try and surface any bugs. First, I’ll go into the workload itself and explain how it can fail when it encounters those bugs. Next, I’ll go into the environment that will exhaustively search through “the multiverse” to trigger and surface those bugs.

The Workload

Let’s start with the workload, for which we’ll use property-based testing. With property-based testing we can encode what should be the invariants of the system. Put simply, given a specific operation: what should the system do and what should it not do. If something unexpected happens it’s a failure of that property, and it fails the workload, just like a test could fail on an assertion.

There are two kinds of properties:

• Liveness/progress properties: a request is allowed to fail, but it should be successful eventually.
• Safety/correctness properties: a certain state/output should never happen.

You can see these properties as test assertions. For example: any successfully published data into a stream should never be lost, and when consuming we should be able to receive all those messages with the same content as they were published. Combine that with scaling and moving the stream while all of that is going on, and if no properties fail you can be confident it all works.

So, we define a workload that (in parallel) performs the following:

• One thread that publishes, in an ordered fashion, into the stream.
• One thread that consumes, in an ordered fashion, from the stream.
• One thread that scales the stream up and down randomly as it sees fit. Occasionally running multiple scale operations back-to-back without stopping, and sometimes waiting to observe that the stream actually settles at the expected replica count.

For ordered publishing we require the sequence in each publish acknowledgement to always increase, with no gaps. That is made possible by a neat property of JetStream’s deduplication window: when re-publishing a message with the same duplicate identifier, we receive the sequence at which that message was persisted. So, if a response to our publish is ever lost, and we retry, we can still expect to receive the correctly ordered stream sequence. For ordered consuming we can use the ordered consumer construct that each client supports, which uses a single-replica ephemeral consumer that’s meant to be recreated when gaps are detected. This allows us to validate end-to-end that any message published is also consumed, no messages lost, and through checksums we know for sure each message consumed equals what we published. Finally, we scale the stream up and down in parallel by issuing stream update requests with a randomized replica count.

Summary:

• The workload exercises ordered publishing and consuming, asserting no data is lost.
• The workload performs stream scaling in parallel, asserting that the stream can scale up and down properly.
• The combination of those asserts that there is no loss of data and no (persistent) loss of availability when performing stream scaling.

Side-note: since the stream will be scaling up and down between replicated and single-replica, the single-replica ephemeral ordered consumer that’s used will also need to scale up and down and move to different servers as desired. This ensures that both the stream and consumer move and scale paths are exercised thoroughly.

The Environment

So, we have a workload that can exercise what we want to be tested, now we only need an environment to run it in. The environment should closely mimic a production setup: we spin up the servers and workload as we would normally on actual hardware across a network. However, if we stopped there then nothing “exciting” would happen; we’d just test the happy path when everything works properly. And, we already know this works since that’s what our regular test suite is supposed to cover. We want to inject as many faults as possible: hard kills, network partitions, increase “the heat”. We want to see if there are any bugs!

We can’t just randomly inject faults though, because then our workload would simply fail: just stop all the servers and don’t bring them back up. Okay, maybe that sounds a little silly… but it does illustrate that we need to bound how long faults are injected. At some point the faults need to be temporarily paused to allow the system to recover. We therefore need to define what “eventually succeeds” means for all the requests that we can make. How long should we keep retrying until we call it a liveness/progress failure?

Let’s define it as retrying for two minutes at most, and guaranteeing that there are at least 30 seconds with no faults. This means that for all the testing we perform here, whatever mess we’ve gotten the cluster into by injecting faults, NATS should be able to recover and succeed quickly, otherwise the test would fail. It might sound like 30 seconds is quite long, but consider everything that might need to happen: we might need to first re-establish closed cluster connections, work through a delayed backlog of messages, a leader election could happen, we’d need to wait for that to settle, some servers might be lagging behind and need to catch up first, and the request is also not retried instantly as you’d need to wait for an error or timeout first and then the retry is also still delayed by at least another second. So, everything considered, it’s actually quite short for a worst-case set of faults that you could throw at the server. Which makes it really significant if an exhaustive set of tests were to fully pass!

As mentioned before, many types of faults can be injected. While you can inject faults yourself by manually hard killing servers or fiddling with the network, automating this makes it more scalable. We’ve built this kind of tooling ourselves as well, where we can inject all kinds of faults and the workload notices if any property would fail as a result. Simpler, but still non-trivial, bugs can be found and fixed this way. But, since this whole process is non-deterministic it will be really hard to develop a fix, re-run the tests and then be confident that it’s truly fixed. It’s also tricky to know exactly which faults to inject, and when, as the hardest bugs may hide behind specific coordinated failures. If we ran this in real time for a bug that only shows up after multiple hours, then we’d need to run the workload and wait for as long as well.

Luckily there are ways around that. You could implement your software in a deterministic way so you can speed up time. Or, you could use a tool like Antithesis, which can run non-deterministic software in a deterministic environment, which lets it speed up time.

Antithesis

So, let’s now actually make it “exciting” and open up “the multiverse”!

I’ve run upwards of 6,000 test hours, which roughly equates to 8 months of “virtual” testing time, within the span of only one month. This testing was performed during the lead-up to the NATS Server 2.15.0 release, and has continued since, with additional improvements already made. Let’s briefly look at a single run of ~400 test hours, showing more than 150 thousand injected faults:

Isn’t it beautiful?

These are all different timelines in “the multiverse”, starting from the top-left, time moving forward to the right, and branching timelines going down. The various colored dots represent different types of faults that were injected in that particular branch. You might notice that each timeline, sometime after the beginning, has an empty “gap” where no colored dots exist. That’s the faults being temporarily paused to allow the system to recover. The faults are toggled on and off for random periods, each lasting at least 30 seconds. Since we currently determine the random fault-on period at the start of the run, each timeline that branches off within that period will have the same initial pause time. After that, each timeline gets its own randomized fault on/off periods, so it quickly looks more “chaotic” afterward.

All of these timelines are separate runs of our workload, all subjected to different kinds of faults at different moments in time. Antithesis effectively fuzzes our workload and tries to find “interesting” state spaces to get the workload into, to see whether that would surface any bugs. Antithesis has all kinds of smart ways to figure out where and when to inject faults, so I won’t get into that here, but you can think of it as: if your workload covers the right code paths, and you let Antithesis run for long enough so it can exhaustively search, then it’s very likely to find any bugs.

This allows for an iterative process of: running the workload in Antithesis, triaging all the findings, then fixing the bugs, and repeating the process until no more bugs are found. This approach helps ensure that the server is thoroughly tested and any potential issues are identified and addressed. Although the testing continued after the 2.15.0 release, we worked hard to make sure the most significant bugs were resolved before releasing a new version.

Development Process

I’ve already covered both the old and new design, and shared some of the details around how we’ve tackled testing it. As we’re slowly coming to a close, I want to share a bit more about the overall process itself. I’ll sketch a timeline of how we went from design, to implementation, to triaging and squashing the remaining bugs. I’ve also linked to the server PRs, so if you’re curious about the nitty-gritty technical details, you can take a look at those as well.

If you’re interested in the overall process and details, feel free to continue reading. If not, you can also skip over this section and on to the Conclusion.

Pre-design

Around February and March 2026, even before NATS Server 2.14 was released, we were already identifying and discussing the general issues of scaling and moving assets, as well as how it relates to peer management and removal. We had already resolved most of the Raft and JetStream replication bugs in prior server releases, given stable membership and replication count, and the “only” remaining piece was the unreliability of scaling and moving under certain conditions. So, we made the decision to focus on fixing “all the bugs” relating to that during the 2.15 release cycle. Especially since this was no simple thing to solve, it needed dedicated time and effort (which may already be clear, but will become clearer as we go through the rest of the process).

Design

We already knew we wanted a desired state reconciliation mechanism, where the meta leader would suggest changes instead of forcing them through, and where the stream and consumer leaders would coordinate to ensure every step of an operation is performed safely. However, as I mentioned before, it all stands or falls on the design itself being solid. A design that allows for shortcuts or can be unsafe given some edge cases will never be able to be fully reliable, so we needed a design that didn’t allow for shortcuts to be made and was safe “by construction”.

Redesigning this part of the server felt a bit like replacing an airplane’s engine mid-flight, or knocking the legs out from under a chair and replacing them so quickly and neatly that you wouldn’t really notice it happened in practice. You also don’t know what you don’t know, so we started with an initial design and got as far as we could with it, fully ignoring any upgrade/downgrade considerations or complications just to find out if the design would hold up and whether anything was missing. This essentially took a couple more months to fully iron out, but we ended up with a solid design with some remaining rough edges that needed to be worked out, and we had some ideas about how to handle upgrading and downgrading in a smooth way.

It would have been too easy to stop there and open one big PR. But it was a laundry list of changes: some could be backported to earlier versions, and splitting the rest into smaller PRs made them reviewable and let us focus on each part separately, knowing exactly where we were heading. Long story short: we re-implemented everything from scratch, but better.

Implementation

Laying the groundwork

Before starting on the desired state work itself, we laid the groundwork with fixes that could be backported, plus some related changes for 2.15.

Stacking desired state PRs

Then we focused on the desired state changes themselves, starting with an initial PR that added the desired state model. Stacked follow-up PRs then rounded out the remaining areas.

Finishing up before releasing 2.15.0

Now that the overall desired state reconciliation is in place, we can start focusing on testing using the workload mentioned before. This is roughly where the month of dedicated testing, triaging, and fixing was spent.

This was also the moment in time where the ADR (Architecture Decision Record) was written up. If you’re interested in reading (even more) technical details, you can find them in ADR-62: JetStream desired state reconciliation.

More fixes after 2.15.0

At some point you need to cut the initial release. We temporarily freeze the code base, which lets everyone dedicate their time to testing what would become the 2.15.0 release, and ensures that we can release it in a timely fashion.

This also let us focus a bit more on testing upgrades and downgrades, ensuring a compatible 2.14 patch release is made available to make upgrading smoother, writing up the various release notes, upgrade guides, blog posts, etc.

Suffice it to say, more improvements could still be made to make it even more reliable, and have been released in later 2.15 versions.

Conclusion

Phew, we’ve made it!

It has been a significant amount of work to get here, but it’s really satisfying to be confident about how much more reliable NATS Server 2.15 is when scaling, moving, and peer-removing.

Thank you for reading! I hope you enjoyed it and learned a bit more about the issues we’ve been working on as well as the design and general process around testing.

If you aren’t on NATS Server 2.15 already, we highly recommend upgrading to enjoy this significant increase in reliability!

To learn more:

• NATS Server 2.15 release announcement
• Upgrading to NATS Server 2.15
• NATS Server v2.15.0 release notes