Claim Check Pattern: Handling Large Messages in Data-Intensive Applications

A message broker is good at moving small messages quickly and reliably. It is usually a poor place to transport a 20 MB export, a PDF, a batch of telemetry, a machine-learning artifact, or a giant JSON document.

That is the problem the Claim Check pattern solves.

Instead of pushing the heavy payload through your queue or event bus, you put the payload in durable storage and send a small reference through the messaging system.

The idea is simple:

That separation becomes increasingly useful as systems become more data-intensive.

Claim Check pattern diagram showing payload stored externally while a reference travels through the message flow

The original Claim Check pattern is described by Enterprise Integration Patterns. The sketch above is reusable under its Creative Commons Attribution terms.

What is the Claim Check pattern?

Imagine checking a suitcase at an airport.

You do not carry the suitcase through every checkpoint, shop, gate, and corridor. You hand it over once and receive a small claim ticket. Later, the ticket lets you retrieve the suitcase.

Claim Check applies the same idea to distributed systems.

A producer has a large payload:

50 MB measurement export

Instead of placing the entire 50 MB inside a queue message, it stores the payload in something such as:

Then the producer publishes a small message:

{
  "jobId": "export-4812",
  "payloadId": "exports/2026/10/export-4812.parquet",
  "contentType": "application/x-parquet",
  "sha256": "...",
  "expiresAt": "2026-10-20T12:00:00Z"
}

The consumer receives this small message, resolves the reference, downloads the payload, processes it, and acknowledges the message.

The queue carries coordination. The object store carries data.

That distinction is the heart of the pattern.

Why large messages become a problem

Messaging infrastructure is normally optimized for high-throughput delivery, retries, routing, ordering, subscriptions, acknowledgements, and dead-letter handling.

Large payloads work against those strengths.

They increase:

They can also run into hard message-size limits.

Microsoft's architecture guidance describes the same core issue: large payloads can exceed broker limits and consume disproportionate messaging resources, reducing throughput for the rest of the workload.

The Claim Check pattern changes the question from:

into:

That is usually the more productive architectural question.

A simple mental model

Without Claim Check:

Producer
   |
   |  10 MB payload
   v
Message Broker
   |
   |  10 MB payload
   v
Consumer

With Claim Check:

                    +------------------+
                    |   Object Storage |
                    |    10 MB payload |
                    +------------------+
                      ^              |
                      | store        | fetch
                      |              v
Producer ---- small reference ----> Broker ----> Consumer

The payload still exists. Claim Check does not magically eliminate bytes.

It removes those bytes from the broker path.

That matters because object storage and message brokers are optimized for different jobs.

What happens to broker traffic?

The following chart uses deliberately simple, illustrative numbers.

Assume 1,000 messages and a roughly 0.5 KB Claim Check reference. It compares only the data carried by the message broker. Payload upload/download through object storage is not included.

Illustrative chart comparing broker-carried data for inline messages versus Claim Check references across different payload sizes

The important point is not the exact number. It is the shape of the system.

With inline messaging, broker traffic grows with payload size. With Claim Check, broker message size stays approximately constant while the heavy data is handled by storage designed for it.

A practical example: asynchronous data export

Suppose a user requests three years of measurements from a data-intensive application.

The export worker creates a 300 MB Parquet file.

A fragile design would try to push the export itself through the queue:

Export worker -> 300 MB message -> queue -> download worker

A better design is:

1. Export worker creates the file.
2. File is written to object storage.
3. Worker publishes a tiny ExportCompleted message.
4. Consumer receives the message.
5. Consumer reads the file when needed.

The event might contain:

{
  "eventType": "ExportCompleted",
  "exportId": "4812",
  "objectKey": "exports/4812/data.parquet",
  "size": 314572800,
  "hash": "sha256:..."
}

Notice what is not inside the message: the actual export.

This architecture also makes the storage layer independently manageable. You can apply lifecycle policies, archival tiers, retention periods, encryption, access control, versioning, or cleanup jobs without forcing those concerns into the broker.

The implementation detail that matters most: store first, publish second

There are two systems involved:

They normally do not participate in one atomic transaction.

That creates an important failure window.

A producer should generally do this:

Write payload -> verify success -> publish Claim Check

Not this:

Publish Claim Check -> hope payload write succeeds

If a consumer receives a reference before the payload is actually available, it has a message it cannot resolve.

But even the safer order has a failure case:

Payload write succeeds
        |
        X producer crashes before publishing message

Now you have an orphaned payload.

That is not a reason to avoid Claim Check. It means cleanup must be part of the design.

For example:

Distributed systems are rarely about eliminating failure. They are about making failure detectable, recoverable, and boring.

The failure modes people forget

This table is more important than the happy-path diagram.

The pattern itself is easy. Lifecycle and consistency are the engineering work.

Do not put long-lived secrets in the Claim Check

A common shortcut is to make the “reference” a long-lived public URL or a powerful storage token.

That couples identity, authorization, and payload location into one string that may be copied into:

A cleaner approach is for the message to carry a stable identifier such as:

exports/4812/data.parquet

The consumer then authenticates to storage independently using its own identity.

In Azure, for example, a worker can use Managed Identity and RBAC rather than receiving a permanent storage secret inside every message.

If temporary delegated access is genuinely required, use a short-lived access mechanism deliberately rather than treating the Claim Check itself as the security model.

Make consumers idempotent

Queues and distributed processing often involve retries.

That means the same Claim Check can be seen more than once.

A consumer should be able to receive:

export-4812
export-4812
export-4812

and still produce one correct business outcome.

Typical techniques include:

Claim Check does not create the duplicate-delivery problem, but large asynchronous workflows make idempotency especially important.

Should every message use Claim Check?

Usually, no.

For a 2 KB domain event, putting the body in object storage would be needless complexity and additional latency.

A practical system can use a threshold:

if (payload.Length <= InlinePayloadLimit)
{
    await bus.PublishAsync(payload);
}
else
{
    var payloadId = await objectStore.PutAsync(payload);

    await bus.PublishAsync(new ClaimCheckMessage(
        PayloadId: payloadId,
        ContentType: contentType,
        Sha256: hash));
}

The exact threshold is workload-specific. It should come from measurement, broker constraints, latency goals, and operational cost — not from a universal magic number.

Your message schema should also make it obvious whether data is inline or external.

Claim Check vs compression vs splitting

These techniques solve different problems.

Compression

Use compression when the payload compresses well and remains comfortably within the broker's operational limits.

It keeps the architecture simple, but the broker still transports the payload.

Split and aggregate

Split a payload when chunks have a meaningful independent structure or can be processed in parallel.

For example, 10,000 independent records may naturally become batches.

But splitting introduces sequencing, correlation, retries, partial completion, and reassembly complexity.

Claim Check

Use Claim Check when the payload is fundamentally one object but the message path does not need to carry the object itself.

Examples:

When Claim Check is a bad idea

Patterns are tools, not commandments.

Avoid Claim Check when:

If a normal message works comfortably, send a normal message.

Complexity should have to earn its place.

A production checklist

Before shipping a Claim Check implementation, I would answer these questions explicitly:

If those answers are clear, the implementation is usually straightforward.

The broader lesson for data-intensive systems

The most useful part of Claim Check is not the code. It is the separation of responsibilities.

A broker is a coordination mechanism.

An object store is a data mechanism.

Trying to make one behave like the other often works at small scale and becomes painful later.

Claim Check keeps the control plane small and lets the data plane use infrastructure designed for large objects.

That is why such a simple pattern continues to matter in modern distributed systems.

When your message starts looking more like a file than a message, that is a strong signal to ask whether the payload should leave the broker path entirely.

Further reading