Claim Check Implementation Ideas

Previously during the Advent of EIP I covered the Claim Check Pattern as part of Day 5: Message Types & Event Payload Strategies. In that write up I left the implementation purposefully simple: write the document to S3 and include the S3 address in the payload for the consuming side to receive and fetch the bytes. This is a pattern I have used time and time again and it works well regardless of whatever object store you use and is very simple for your team to reason about.

Recently I worked with a team that wound up having an interesting problem… multiple downstream services needed to fetch the payload specified by an event type. The simple solution was to wrap the details of fetching the payload in a library to encapsulate the bucket layout structure and replicate the IAM permissions to read from the bucket to each consuming service.

The Gap: Tight Coupling with Object Storage

It worked, but I didn’t really like it. It was really bugging me for a few reasons:

• IAM Sprawl - any new service utilizing this library would also need to replicate IAM permissions for reading the bucket. Implementation becomes a library addition + an IAM permission add on
• Migration Coordination - any changes to the object structure would need to be rolled out in coordination with consuming services, some owned by different teams. Achievable, but felt like it would be pretty heavy-handed!
• Encapsulation at the Wrong Layer - Wrapping the access patterns was better than nothing, but felt like the same smell I have seen in my career where many services share the same database via a library of ORMs. Well intentioned, but messy. Long ago we solved that problem by having services access shared data via an API to provide better abstraction over shared libraries and this felt like repeating that old mistake.

While this all sounds great technically, this tight coupling has a tangible end-user cost… evolution of architecture to fit new services and features come at a heavy cost due to the amount of coordination and mitigation of impacts involved. I’ve seen many projects like this over my career and they tend to draw out way longer than they need to.

Solution: Encapsulate the Document Store with a Gateway

I’ve already touched on what the solution here is while talking about the database analogy: there is a preference to wrap a database in an API rather than letting multiple services hit it directly. The object store for claim-checked documents is a database, so we should give it the same treatment. For the write side I won’t talk about that today, we can assume that the application writing the claim check owns the data and as a result is allowed to have intimate access to the storage. Let’s focus on the read side instead.

The first approach one might go after on this is to simply wrap the object store in an API, and that isn’t a terrible idea, it is perfectly fine for most use cases. However the appetite to build and own a new service can be low for some teams, so why not rely on something purposefully built for this scenario? nginx-s3-gateway can be a great fit here. It’s already a battle-tested solution and provides some other interesting features like caching for frequently retrieved objects.

The architecture at this point becomes effectively an API that consumers talk directly to using a common identifier that the publisher provides as the claim check key. The power of using nginx-s3-gateway becomes more relevant when you factor in the caching layer it provides, since if six consumers retrieve the same object, only the first request hits S3 and the rest are served from cache.

The caching is layered in memory as part of nginx, so if you are running a fleet of multiple instances you will want to keep that in mind. You’ll still benefit from some caching, but not every identical request will hit the cache. If you DO need a shared cache, you might look into some other options such as OpenResty with Redis cache access.

Bonus: The Route Table as a Migration Lever

There are a lot more configuration options you can put in place here, even using something like Envoy to control the route and then completely change the backing store to something besides straight S3 reads if you desire! Once something like Envoy is sitting in front of the read path, the route table stops being just a security boundary and becomes the place where you decide how a claim check gets resolved, not just whether it’s allowed. You could match on the path, a header, or something encoded in the identifier itself, and quietly send different slices of traffic to different places: most objects stay on S3, a specific tenant’s objects live somewhere else, anything older than N days gets served from a cold tier, or the whole thing fronts a store that isn’t S3-shaped at all. None of that requires touching a single consuming service, because as far as they know they’re calling the same endpoint with the same reference they’ve always had.

That’s what makes this kind of migration so much less painful than the alternative. Moving off a storage provider, splitting hot and cold data, or trialing a competing object store without fully committing to it can all happen behind the route table, a slice of traffic at a time, with a rollback that’s just changing a route instead of coordinating a synchronized deploy across every team that reads claim checks. How powerful is that, to be able to change the object store without any changes to the consuming services at all?

When This Doesn’t Fit

This pattern won’t always fit your use case. If you have a simple application using messaging with claim checks for backend processing this might be overkill. The word “simple” here does a lot of heavy lifting though. Once your application reaches a certain scale where the publisher and consumers are complex enough there might be some benefit to introducing a gateway in front of your object storage, especially if the backend processors are split off to their own codebases and deployed separately. But a gateway for something like a 15-line Oban processor might simply be a bit of overkill.

See It In Action

I always like to say, seeing is believing so I have put together a “small” demo that shows an implementation of nginx-s3-gateway in front of a localstack S3 bucket and uses Envoy for the routing. This also uses NATS for the messaging layer with a 256KB cap. This demo also includes some features such as authenticating with the gateway and uses a series of justfile tasks to run the demo, including some pre-baked scenarios to see it in action.

The demo illustrates a gateway client that uses authentication with very straightforward construction:

const gateway = new GatewayClient({
  baseUrl: process.env.OBJECT_GATEWAY_URL,
  getAuthHeader: async () =>
    `Bearer ${await mintServiceToken({ privateKey, kid: KID, subject: SUBJECT_NAME, scopes: ["objects:read"], tenant: TENANT })}`,
  timeoutMs: 5_000,
});

And the consumer loop leverages a single line to fetch payloads from the gateway.

const event = m.json<ClaimCheckMessage>();
// The entire authorization and retrieval story, from this service's point of view.
const object = await gateway.get(event.ref);

processed.set(event.messageId, {
    messageId: event.messageId,
    ref: event.ref,
    filename: event.hints.filename,
    declaredSize: event.size,
    resolvedSize: object.size,
    resolvedAt: new Date().toISOString(),
});

That’s it! You can clone it and explore it at your leasure out at jamescarr/object-gateway-demo. Enjoy!