The Route Table Is the Contract: Claim Checks Behind a Gateway

Recently I helped a team where several services all needed to read documents that another service wrote to an S3 bucket. We wanted to keep the object store, the bucket layout, and the IAM permissions independent from the consumers, so the owning team could change any of them without asking anyone else’s permission. The first pass was the obvious one: a shared library that wrapped the details of fetching a document, encapsulating the bucket layout and replicating the IAM read permissions out to every consuming service.

It did the job. But the argument I want to make isn’t that this access should have been denied. It’s that there should have been nothing to reach in the first place. A policy that says no can be misconfigured, forgotten, or scoped too broadly by whoever wrote the Terraform. A network with no path at all cannot. I’ll prove that difference with a running demo at the end of this post, but keep it in your pocket while you read the rest: it’s the strongest version of everything that follows.

Working through the implementation, the shape of the problem kept looking familiar. One service writes a document somewhere durable, a small reference travels, and other services exchange that reference for the bytes later. That is the read half of the Claim Check Pattern, which I covered during the Advent of EIP as part of Day 5: Message Types & Event Payload Strategies. In that write up I left the implementation purposefully simple: write the document to S3 and include the S3 address in the payload for the consuming side to receive and fetch the bytes. It’s a pattern I have used time and time again, it works with whatever object store you have, and it is very simple for your team to reason about. It also has exactly the coupling the team above was living with, so for the rest of this post I’m going to treat them as the same problem.

The write side is the half that usually needs no defending, and it is worth watching before we get to the part I want to argue about. Documents of wildly different sizes go in; the same 312-byte message comes out every time. Switch to the nightly batch and every single document is past the broker’s 256KB cap, which is the version of this where nobody had a choice.

The Gap: Tight Coupling with Object Storage

The shared library worked, but it was really bugging me for a few reasons, roughly in order of how much they should bother you:

• Migration Coordination - any change to the object structure needs a coordinated rollout across every consuming service, some of them owned by other teams. Achievable, but heavy handed for something that should be an internal decision.
• Blast Radius - wrapping the access pattern in a library still means every consuming service ends up holding real credentials to the bucket. It’s the same smell I’ve seen in services that share a database through a library of ORMs: well intentioned, but it means a bug or a leak in any one consumer is a bug or a leak in the bucket itself. We fixed this for databases by putting an API in front of them. This is the same mistake wearing a different accessory.
• IAM Sprawl - every new consumer needs its own read policy against the bucket. This is the smallest of the three. A Terraform module makes it a one-line addition, and a skeptical reader would be right not to weigh it heavily. It’s a real chore. It just isn’t the argument that should carry this post.

Each of these is survivable on its own. Together they have a real cost for anyone downstream of the decision: evolving the architecture to fit new services and new features comes at a heavy price, because every change means coordinating across teams and reasoning about the blast radius of getting it wrong. I’ve seen projects shaped like this over my career, and they tend to run far longer than they need to.

Why Not Presigned URLs?

Before getting to the gateway, it’s worth answering the question every experienced reader is already forming: why not just hand out a presigned URL? It’s one line in most SDKs, and it means the payload bytes never have to go anywhere near your own infrastructure. That’s a reasonable instinct, and for the right shape of problem it’s the correct answer. It just isn’t the answer for a claim check, and the reasons are worth being specific about.

Presigned URLs and events have opposite lifetimes. An event on a durable stream is meant to be replayed, whether that’s from JetStream retention, a dead letter queue someone is retrying by hand, or a new consumer backfilling from the start of the topic. A presigned URL is a credential with an expiration stamped into the query string: capped at seven days if you sign it with a long-lived IAM user, and often far shorter than that if it’s signed with the temporary credentials a role actually hands your service in production. Put one in an event and you’ve written a time bomb into your own message history. Replay it a week later and every consumer gets a 403, even though the object it points to is sitting untouched in the bucket. A URN and a digest don’t have this problem, because they don’t expire. They’re just as resolvable a year from now as the moment they were published. This is the argument, not a footnote to it.

A presigned URL is also a bearer capability, which means whoever holds it can use it, full stop. It leaks into application logs, into distributed traces, into the payload of a dead letter queue someone dumps into a support ticket. There is no way to ask afterward whether it was consumer A or consumer B who fetched the bytes, no way to revoke it short of rotating the key that signed it, and no audit trail beyond whatever the object store’s own access logs happen to capture. A design where the answer to “who read this” is “anyone who had the URL” doesn’t have per-consumer authorization at all. It just looks like it does from a distance.

The URL also encodes the store. Bucket name, region, endpoint, all of it baked into the string the moment it’s minted. The migration lever I’ll get to shortly, where you move providers or split hot and cold storage behind a route table without touching a single consumer, doesn’t survive this. Every presigned URL still in flight names the old bucket, and there’s nothing you can do about the ones already sitting in a queue.

And there’s no shared cache to speak of. A presigned URL is minted for one request, so there’s no place in front of the store where six consumers of the same object could ever collapse into one request the way they do behind a caching proxy. Every read is a fresh round trip to the origin, no matter how many times the same bytes get asked for.

The one advantage presigned URLs actually have is real, and worth keeping: the payload bytes never transit your own infrastructure. If that matters to you more than everything above, there’s a middle path that keeps most of it. The gateway still owns the stable reference, still authenticates the caller, still checks the tenant, still writes an access log entry, and instead of proxying the bytes itself it answers with a 302 to a presigned URL it mints on the spot with a lifetime measured in seconds. The reference in the event never expires. The credential handed out at the very end of that request does, and it’s gone by the time anyone could do anything useful with it.

Which gets at the real point: an endpoint that exchanges a stable reference for a freshly minted, narrowly scoped, short-lived credential is a gateway. The identity check, the tenant check, and the route table are still there, they’ve just moved to the front of the request instead of being embedded in a URL. Most attempts to avoid running a gateway by handing out presigned URLs end up building one anyway, with worse ergonomics and none of the audit trail.

Solution: Encapsulate the Document Store with a Gateway

I’ve already touched on what the solution here is while talking about the database analogy: there is a preference to wrap a database in an API rather than letting multiple services hit it directly. The object store for claim-checked documents is a database, so we should give it the same treatment. For the write side I won’t talk about that today, we can assume that the application writing the claim check owns the data and as a result is allowed to have intimate access to the storage. Let’s focus on the read side instead.

The first approach one might go after on this is to simply wrap the object store in an API, and that isn’t a terrible idea, it is perfectly fine for most use cases. However the appetite to build and own a new service can be low for some teams, so why not rely on something purposefully built for this scenario? nginx-s3-gateway can be a great fit here. It’s already a battle-tested solution and provides some other interesting features like caching for frequently retrieved objects.

The architecture at this point becomes effectively an API that consumers talk directly to using a common identifier that the publisher provides as the claim check key. The power of using nginx-s3-gateway becomes more relevant when you factor in the caching layer it provides, since if six consumers retrieve the same object, only the first request hits S3 and the rest are served from cache.

That caching story only works because of a detail from the write side: each claim check gets a fresh address the moment it’s written, and once minted, an address is never reused for different bytes. Nothing under a given key ever changes, so there’s no invalidation problem to solve. The cache doesn’t need to know when to forget something, because nothing it holds ever goes stale.

That last sentence is easy to write and easy to nod along to, so here it is running. Six readers, one object is the scenario it describes, and it is honest about the detail I glossed over: the very first requests all miss, because they are in flight together before any of them has finished filling the cache. Once one lands, the store goes quiet and stays quiet. If that opening burst matters to you, it is what proxy_cache_lock is for.

Without the cache is the scenario worth a minute of your time, because it separates the two jobs the gateway is doing. Consumers still hold no credentials with caching switched off. Encapsulation and cache hit rate are independent wins, and only one of them is a performance argument.

The caching is layered in memory as part of nginx, so if you are running a fleet of multiple instances you will want to keep that in mind. You’ll still benefit from some caching, but not every identical request will hit the cache. If you DO need a shared cache, you might look into some other options such as OpenResty with Redis cache access.

None of this is free, and it’s worth being upfront about what you’re taking on. The gateway becomes a new dependency on the read path: if it’s down, every claim check is unreadable, regardless of whether the object store itself is fine. It’s an extra network hop on every read that didn’t exist when consumers hit S3 directly, and nginx is now a proxy sitting between every consumer and every byte they retrieve, which puts its own throughput and connection limits into your capacity planning. None of these costs are hidden or unusual, they’re the same costs any API gateway or reverse proxy takes on, but they’re real and they belong in the decision, not left for someone to discover during an incident.

Bonus: The Route Table as a Migration Lever

There are a lot more configuration options you can put in place here, even using something like Envoy to control the route and then completely change the backing store to something besides straight S3 reads if you desire! Once something like Envoy is sitting in front of the read path, the route table stops being just a security boundary and becomes the place where you decide how a claim check gets resolved, not just whether it’s allowed. You could match on the path, a header, or something encoded in the identifier itself, and quietly send different slices of traffic to different places: most objects stay on S3, a specific tenant’s objects live somewhere else, anything older than N days gets served from a cold tier, or the whole thing fronts a store that isn’t S3-shaped at all. None of that requires touching a single consuming service, because as far as they know they’re calling the same endpoint with the same reference they’ve always had.

That’s what makes this kind of migration so much less painful than the alternative. Moving off a storage provider, splitting hot and cold data, or trialing a competing object store without fully committing to it can all happen behind the route table, a slice of traffic at a time, with a rollback that’s just changing a route instead of coordinating a synchronized deploy across every team that reads claim checks.

When This Doesn’t Fit

This pattern won’t always fit your use case. If you have a simple application using messaging with claim checks for backend processing, this might be overkill. The word “simple” here does a lot of heavy lifting though. Once your application reaches a certain scale, where the publisher and consumers are complex enough, there’s real benefit to a gateway in front of your object storage, especially once the backend processors are split off into their own codebases and deployed separately. A gateway in front of a 15-line Oban processor is a bit of overkill.

It can also be too hot a path. The gateway is a new availability dependency sitting on every read, an extra network hop that didn’t used to be there, and nginx is now streaming every payload byte your consumers ever ask for. If your objects are large, your reads are latency sensitive, and the fan-out this whole design exists to solve doesn’t actually apply to you, the 302-redirect hybrid from earlier or skipping the gateway entirely are both more honest choices than bolting one on because it’s the pattern of the moment.

And there’s a case where presigned URLs are simply the right answer: one consumer, reading the payload within minutes of it being produced, with no replay story and no second team that will ever need the same reference. That is most of the claim checks most people will ever write. This whole post is about what changes once that stops being true.

See It In Action

Everything above is easier to accept when it’s running. I put together a demo: nginx-s3-gateway in front of a localstack S3 bucket, Envoy doing the routing and authorization, NATS for messaging with the same 256KB cap Amazon SQS has, and a set of justfile tasks that stand the whole thing up and run through a series of scenarios against it.

You need Docker and Node 24 or newer. There is no AWS account involved, since LocalStack stands in for S3. The clone below pins the read-side-post tag, the exact state of the repo this post describes; main has since grown a write-side gateway too, which is where the follow-up post picks up:

git clone --branch read-side-post https://github.com/jamescarr/object-gateway-demo
cd object-gateway-demo
pnpm install
just up      # build and start everything, wait until it is healthy
just demo    # run the scenarios

just demo runs ten scenarios against the live stack. Each one is a claim the design makes that you would otherwise have to take on trust.

The demo illustrates a gateway client that uses authentication with very straightforward construction:

const gateway = new GatewayClient({
  baseUrl: process.env.OBJECT_GATEWAY_URL,
  getAuthHeader: async () =>
    `Bearer ${await mintServiceToken({ privateKey, kid: KID, subject: SUBJECT_NAME, scopes: ["objects:read"], tenant: TENANT })}`,
  timeoutMs: 5_000,
});

And the consumer loop leverages a single line to fetch payloads from the gateway.

const event = m.json<ClaimCheckMessage>();
// The entire authorization and retrieval story, from this service's point of view.
const object = await gateway.get(event.ref);

processed.set(event.messageId, {
  messageId: event.messageId,
  ref: event.ref,
  filename: event.hints.filename,
  declaredSize: event.size,
  resolvedSize: object.size,
  digestVerified: object.verified,
  resolvedAt: new Date().toISOString(),
});

That is the whole story from the consuming side. No S3 client, no bucket name, no region, no IAM policy. The single piece of configuration that gave this service access to shared payloads is OBJECT_GATEWAY_URL.

What it holds instead is the reference the publisher put on the queue:

urn:demo:object:v1:acme:019fe41a-a634-7b10-b123-c4a483e5b332:sha256-qSHbzFOWgz2gLZMNYrcmimKWSy1XpsV9ZnJX5hza3rA
                  └tenant┘ └──────────── address ───────────┘ └──────────────── integrity ────────────────────┘

The tenant and the address say what to ask for. The digest lets the reader verify what it got without having to trust the store, the gateway, or the network in between, and it never leaves the consumer: the gateway has no use for it. The tenant isn’t just a label either. The authorizer checks it against the tenant claim on the caller’s own token before the request ever reaches nginx, so a valid token for one tenant asking for an object under another tenant’s namespace is refused regardless of whether the reference itself is correct.

Coming back to the route table being the contract, here is the read path in its entirety:

- name: read_object
  match:
    safe_regex:
      regex: "^/objects/[a-z0-9][a-z0-9-]{1,31}/[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$"
    headers:
      - name: ":method"
        string_match: { exact: "GET" }
  route:
    cluster: object_read_gateway

- name: deny_by_default
  match: { prefix: "/" }
  direct_response:
    status: 404

There is no way to ask that gateway for a listing, a traversal, or a key that is not a UUIDv7 under a tenant, and everything it does not recognize gets the same uninformative 404. That’s deliberate: a wrong path and a right path that was refused look identical from the outside, so probing the gateway for its shape gets you nothing back. Pointing object_read_gateway at a different cluster is the migration lever I was describing earlier, and it is a one line change that no consuming service ever notices.

This is the promise from the top of this post, paid off with a running example instead of an assertion. My favorite scenario is the third one, because there is a real difference between a policy that denies access and there being no access to deny:

3. The consumer cannot reach the object store at all
  ✔ holds no AWS credentials
  ✔ knows no bucket name, region or endpoint
  ✔ is not attached to the bucket-facing network: that network holds only object-gateway, producer, localstack
  ✔ while the producer, which owns the documents, is
  ✔ cannot even resolve the object store's hostname: localstack: ENOTFOUND

Notice where that boundary falls. It is drawn around ownership rather than around applications in general. The publisher owns the documents it writes, so it talks to the bucket directly and sits on that network, exactly as I assumed at the top of this post. Every service that merely needs to read what it was sent goes through the gateway. Whether the write side deserves the same treatment is a longer conversation, and it’s the subject of the follow-up post.

Two other tasks are worth running while the stack is up. just watch-gateway follows Envoy and the authorizer making their decisions live, and just network-report prints exactly which containers can see the bucket at all.

You can clone it and explore it at your leisure at jamescarr/object-gateway-demo, tagged read-side-post.