Turning StreamingInside Out
A new architectural paradigm for modern event streaming — how messaging-first design eliminates the complexity, cost, and constraints of traditional streaming platforms.
Executive Summary
Traditional event streaming platforms are storage-first. The broker's identity is a distributed, partitioned commit log, and every other capability — routing, filtering, caching, state, integration — is assembled around it from external components. That arrangement works, but it fixes the cost of change: partitions cannot move freely, clients must know the cluster topology, and each added component is another dependency to operate and another way to fail.
This paper argues for inverting that relationship. Put messaging at the foundation, make storage a pluggable backend, and address data through a hierarchical subject namespace rather than fixed partitions. Streaming, key-value, and object semantics then become composable interfaces over one substrate — enabled by configuration rather than by deploying more infrastructure.
The inside-out model at a glance
- 4
- APIs — stream, key-value, object, and pub-sub — served by a single system instead of four separate vendors.
- 1
- Binary to operate. No ZooKeeper, no BookKeeper, no MirrorMaker, no external cache.
- 0
- Partition rebalancing. Nodes join a cluster and are immediately useful, with no downtime.
- ∞
- Consumer granularity. Subjects replace fixed partitions as the unit of parallelism.
01
The problem
Traditional Streaming Hits a Ceiling
Traditional platforms are storage-first: the broker is fundamentally a distributed commit log, and everything else — routing, filtering, caching, processing — is bolted on from outside.
That design served a decade of datacenter-scale workloads well. But the coupling it assumes now shows up as friction: capabilities that belong near the data live in separate systems, the unit of parallelism is fixed before the first event is ever written, and expanding beyond a single region means introducing more infrastructure to manage.
- Storage welded to the broker
- Events are written chronologically to disk and the broker’s identity is fundamentally a distributed commit log. Coupling storage to the broker makes the system harder to adapt to new use cases.
- Capabilities pushed outside
- Routing, filtering, caching, and processing are handled by external components or custom application code — each adding a network hop and latency.
- Rigid partitioning
- Partition count is fixed at topic creation and cannot change dynamically without risking order-guarantee violations. It also caps how many consumers can work in parallel.
- Mandatory dependencies
- Kafka historically required ZooKeeper and, even after KRaft, a production deployment typically still runs Kafka Connect, Schema Registry, and MirrorMaker 2 as separate JVM processes; Pulsar requires BookKeeper and ZooKeeper. Each is additional expertise, additional capacity, and an additional point of failure.
- LAN-scale by design
- Traditional systems scale well inside a datacenter, but global topologies need dedicated intermediaries like MirrorMaker, with their own dependencies.
- Leaky internals
- Engineers must understand partition management, offset management, and consumer group management before they can ship a feature.
The platform ends up dictating the application architecture, rather than adapting to it.
02
The new paradigm
Turning the Model Inside Out
Turning something inside out means changing how it is organized. Applied to streaming, it means asking which capabilities belong inside the broker — and which never did.
Invert the model: make messaging the foundation, treat storage as a pluggable backend, and address everything through a subject hierarchy rather than fixed partitions. Four pillars hold the architecture up.
- 01
Messaging
Building streaming on a messaging layer provides both streaming and messaging semantics under one umbrella. External messaging systems are no longer required for high fan-out, and co-locating these capabilities dramatically improves performance at scale.
- 02
Storage
Storage becomes a pluggable backend — object stores, databases, specialized log stores — reached through well-defined interfaces, eliminating the tight coupling between where events live and how they are processed.
- 03
Composability
Topologies are composed from loosely coupled services communicating through well-defined event contracts and APIs, rather than configured inside a monolithic platform.
- 04
Addressing
Subject-based semantics become the addressing paradigm: a token-delimited destination combined with a powerful wildcard subscription model.
Addressing is where the shift becomes concrete. A subject is a token-delimited destination, and consumers subscribe with wildcards — so one stream serves global, regional, and per-product consumers at once, none of them aware of partitions.
- orders.>
- Receives all order events
- orders.georgia.atlanta.>
- Receives only Atlanta orders
- orders.*.*.*.product_7
- Receives product 7 orders globally
03
Consequences
What the Inversion Unlocks
Once messaging is the substrate and storage is pluggable, a series of hard constraints stop being constraints. The full paper devotes a chapter to each; here is the short version.
- 01
Four APIs, one system
Key-value and object stores are built on top of streams rather than shipped as separate products. A bucket is a stream with specialized message handling; objects are chunked into messages and reassembled on read. Messaging, streaming, KV, and object storage stop being four vendors.
- 02
Routing as configuration
Subject hierarchies and wildcards move routing and filtering out of application code and into configuration — no external stream processor, no extra network hop. Coarse streams can be refined into finer ones, and many sources merged into one, by declaring subject mappings.
- 03
Elastic scaling
A new node is recognized immediately and can accept client connections without a rebalance. Stream replicas expand dynamically with zero downtime, and because storage is decoupled from the streaming layer, compute and storage scale independently.
- 04
Unbounded granularity
Consumer parallelism is no longer capped by partition count. Subjects give an effectively unlimited number of independent consumption units, so the platform stops dictating how finely you can slice your workload.
- 05
Client simplicity
A client connects to any server in the cluster and names the subject it cares about. No broker URLs per partition, no partition IDs, no offset bookkeeping — subject-based routing dispatches transparently.
- 06
Global by default
Regional clusters connect directly through internal gateways to form a mesh within and between clusters. Expanding from one region to many is a matter of configuring gateway destinations.
04
Side-by-side
Traditional vs. Inside-Out
The distinction that matters: whether the platform dictates your application architecture, or adapts to it.
| Traditional model | Inside-out model |
|---|---|
| Platform dictates application architecture | Platform adapts to application needs |
| Storage-first architecture (partitioned log) | Messaging-first architecture |
| Tightly coupled storage | Pluggable, decoupled storage |
| Rigid — consumer groups only | Composable — KV store, object store, pub-sub, stream |
| Fixed partitions as the scaling unit | Dynamic, fully elastic scaling |
| Routing via partition keys and custom logic | Routing via subject hierarchy — config-driven |
| Consumers limited by partition count | Consumers effectively unlimited |
| Complex operational model | Simple operational model |
| Regionally bound; extra components for geo-scale | Natively globally scalable — edge to cloud, no extra components |
| Flexibility comes at the cost of complexity | Flexibility is a native architectural property |
External dependencies
None required
Consumer parallelism
Effectively unlimited
Geographic scale
Edge to cloud, native
05
The full argument
Get the Complete White Paper
The paper works through the architecture chapter by chapter — with figures, configuration examples, and the full comparison against traditional platforms.
Inside the paper
- Traditional streaming architecture and its six structural shortcomings
- The four pillars of the inside-out model, in depth
- Composability: KV and object stores built on streams
- The three layers of decoupling — storage, endpoint, access pattern
- Routing, filtering, splitting, and merging via subject mappings
- Cluster, multi-region, and storage scaling dimensions
- Operational model and total cost of ownership analysis
- Author
- Bruno Baloi
- Version
- V 1.0, 2026
- Format
Faster time-to-value
Through pattern reuse, composability, and the elimination of external dependencies.
Lower operational costs
No additional components required for scaling, regionally or globally.
Revenue growth enablement
Rapidly expand streaming infrastructure to new geographies via native super-cluster configuration.
06
Common questions
Frequently Asked Questions
Direct answers to the questions engineers and architects ask most often about inside-out streaming, NATS JetStream, and how both compare to partitioned platforms like Kafka.
