Reading large NATS Object Store objects slowly

A community member asked how to process a large object from a NATS Object Store over many minutes without buffering the whole object in the client, after seeing an error like:

1
nats: slow consumer, messages dropped on connection

This can happen when the application receives object chunks faster than it processes them. For example, downloading a 500 MB audio file and doing CPU-heavy signal processing inline can make the client behave like a slow consumer.

Short answer

The high-level Object Store Get API is intended to download the object as quickly as possible. It has flow control, but it is not currently designed as a client-paced, long-running processing API with knobs to control the rate of object chunk delivery.

If you expect to process an object slowly over many minutes, consider dropping down to the underlying JetStream stream APIs and reading the object’s chunk messages yourself. That gives you more control over how quickly you fetch chunks and, if using a consumer, over consumer settings such as acknowledgement behavior and ack wait.

This is still JetStream. You are not dropping down to Core NATS messaging; you are bypassing the higher-level Object Store convenience API for this read path.

Why the Object Store Get API may not fit

NATS Object Store is built on JetStream. Objects are represented as metadata plus chunk messages in an underlying JetStream stream.

The high-level Object Store Get operation is convenient when the client wants to retrieve the object normally. But if your application reads a little data and then spends a long time processing before reading more, the server and client library may continue trying to move data efficiently. If the client cannot keep up, you may see slow-consumer behavior.

This is especially relevant when:

• the object is large,
• processing each chunk is expensive,
• the total processing time is much longer than the expected transfer time,
• you do not want to buffer the whole object locally, and
• you need the application to decide when to request the next chunk.

Recommended approach: use JetStream directly for this read path

For slow, incremental processing, use the lower-level JetStream APIs against the stream that backs the object store.

At a high level, the flow is:

1. Read the object store metadata for the named object.
2. Use that metadata to identify the internal object identifier used for the object’s chunk subject.
3. Create or use a JetStream read path for those chunk messages.
4. Fetch chunks incrementally.
5. Process each chunk before requesting more.

One practical way to understand the exact API calls for your bucket and object is to run:

nats obj get <bucket> <name> --trace

The trace output shows the operations the CLI performs, including the metadata lookup and the chunk subject used for the object data. Use that as an inspection tool rather than hard-coding assumptions without verifying them in your environment.

Option 1: Pull consumer filtered to the object chunks

A pull consumer is often the natural fit when your application wants to control pacing.

Instead of having the object data pushed to the client as fast as the Object Store Get path allows, your application explicitly fetches a small batch, processes it, acknowledges it, and then fetches the next batch. Because nothing is delivered until the application asks for it, the read rate is bounded by your processing loop rather than by the server, which is what keeps the client from becoming a slow consumer.

Conceptually:

1
lookup object metadata
2
identify chunk subject
3
create pull consumer filtered to that chunk subject
4
loop:
5
  fetch one or a few chunk messages
6
  process chunk data
7
  ack messages when appropriate

This gives you control over the consumption rate. It also lets you tune consumer settings that matter for long processing, such as ack wait, depending on the behavior you want.

Tradeoffs to consider:

• You are using lower-level JetStream mechanics instead of the Object Store convenience API.
• A consumer filtered to the object’s chunk subject delivers chunks in stream order; make sure your processing loop preserves that order when reassembling the bytes.
• You need to decide when to acknowledge messages. If processing each chunk can take a long time, make sure the consumer’s ack wait is compatible with that processing time, or design the loop so work completes before the ack deadline.
• You should keep the fetch batch size small if the goal is to avoid buffering a large amount of object data.

Option 2: Direct gets from the underlying stream

Another lower-level approach is to use JetStream direct get requests against the stream that backs the bucket. A direct get is a request/reply operation: the application asks for a specific message and receives it in the reply. Because there is no push subscription delivering data on its own, the client fully controls when each fetch happens, so the slow-consumer condition does not arise.

This works well when your application can address the chunk messages it needs. The object metadata records the object’s total size and chunk count, and all of an object’s chunks share a single subject and are stored in order, so the application can request them in sequence at whatever rate its processing allows.

Tradeoffs to consider:

• Direct get must be enabled on the underlying stream. You can confirm this from the stream’s configuration before relying on it.
• Your application is responsible for requesting the correct messages and reassembling the chunks in the right order.
• There is no consumer-side acknowledgement or redelivery; each get is an independent request, so your application handles retries itself.
• Depending on how you structure the read path, a pull consumer filtered to the object’s chunk subject may be simpler for straightforward sequential processing.

Is this still Object Store?

It is still the same data stored by NATS Object Store, and it is still JetStream underneath. The difference is which API layer you use for reading:

• Object Store API: convenient, optimized for retrieving the object normally, but not currently designed with rate-control knobs for very slow client-managed processing.
• JetStream stream API: lower-level, more application code, but more control over fetch pacing and consumer configuration.

For writes, metadata management, and ordinary reads, the Object Store API remains the simpler interface. For a special long-running processing path, using JetStream directly can be a better fit.

Practical guidance

If your application needs 30 minutes to process a 500 MB object, avoid coupling processing directly to a high-level Get call that tries to download the object quickly.

Instead:

1. Use the Object Store metadata to locate the object’s chunk messages.
2. Use a pull consumer filtered to those chunks, or direct get requests, to read chunk messages at your own pace.
3. Fetch only as much as you are ready to process.
4. Tune consumer acknowledgement settings for your processing model.
5. Keep the Object Store API for the workflows where fast whole-object retrieval is appropriate.

This approach trades convenience for control. For large objects with slow per-chunk processing, that control is often what prevents the client from becoming a slow consumer while also avoiding a full-object buffer in application memory.

Want help from the NATS experts? Meet with our architects to get help tailored to your use case and environment.