Give the Writer a Gateway of Its Own

The last post put a gateway in front of the read side of a claim check and left the write side alone on purpose. The producer owned the documents it wrote, so it held s3:PutObject and wrote to the bucket directly, and I said that was a fine decision for exactly one writer. I closed with a line I’ve been meaning to come back to: whether the write side deserved the same treatment was a longer conversation. This is that conversation, and this time it comes with a second commit history to point at instead of just an argument.

The short version of what changed: the producer no longer holds a bucket credential. It holds the objects:write scope and calls the same gateway a reader would, on a different route. A new service called ingest is the only thing that touches S3 to write now, the same way the read-side gateway was already the only thing that touches S3 to read. And because both directions go through a gateway, I could do something to the write side I couldn’t have done to the direct-write version: split one tenant’s objects into their own bucket without shipping a change to the producer or the consumer. That part is the actual payoff, and it’s worth reading the boring parts to get there.

What Direct Writes Cost You

The case for leaving the writer alone rests entirely on it being one service that owns the data. That was true for the demo in the last post, and it’s true for plenty of real systems, which is why I want to be honest about exactly when it stops being true rather than pretending it never was.

A second writer means a second copy of the bucket layout. storageKey was shared code, but the decision to call it, use it correctly, and set the same conditional-write header lived in whichever service happened to write. That’s fine right up until a second team writes a second producer, and now the tenancy convention is something two codebases have to agree on instead of one gateway enforcing it.

Credential sprawl restarts on the write side. The read side went from N services holding s3:GetObject down to one. Every new writer undoes half of that: it’s back to N services each needing s3:PutObject, a bucket name, a region, and an endpoint, the exact sprawl the read-side gateway existed to kill.

Retries have nobody to arbitrate them. A writer that mints its own address and times out waiting for the store’s acknowledgement has no way to know if the write actually landed. Retry, and a timeout that actually succeeded leaves two objects and an orphan. There’s no third party in that exchange, just the writer and the store, so nothing catches it.

Nothing has a lifetime, and nobody can delete anything. Claim-check payloads almost always have a natural expiry tied to the broker’s retention window, and every object written directly lives forever because the writer that knew the right lifetime has nowhere to say so. There’s no delete route either, no scope, no story, which is the one that blocks adoption outright for anything touching customer data.

None of this means the direct-write version in the last post’s demo was wrong. It’s the list of things that start mattering the moment a second team starts writing claim checks, which is exactly the point at which standardizing the write path stops being premature.

Three Ways Bytes Can Reach the Store

Once you decide the write side needs the same encapsulation the read side got, there’s a real fork, and it’s worth being explicit about all three branches instead of jumping straight to the one I built.

B is the tempting one because it takes the gateway off the data path entirely: mint an address, hand back a presigned URL, and let the writer PUT straight to the store. It also throws away the exact property that made the read side worth building, that the caller cannot reach the store at all. If a writer needs to resolve the store’s hostname to do its job, “cannot reach it” stops being true for writers, and that’s not a detail, it’s the whole argument.

C keeps the isolation and drops the application code from the byte path, using Envoy’s aws_request_signing filter with use_unsigned_payload: true so the gateway signs the request without buffering the body. It’s a legitimate answer for large payloads, and it’s still on my list, but it means two round trips for every write, and integrity in flight ends up resting on TLS plus a client-declared digest instead of a gateway that watched every byte go by.

A is what I built first, and the reason isn’t that it’s simplest, though it is. It’s that a writer proxying bytes through ingest ends up exactly as isolated as a reader already was: no credentials, no bucket name, no route to the store at all. That’s the property everything else in this design is protecting, and it’s the one B trades away for bandwidth. What A costs is a hard ceiling on payload size well under the 5 GB a raw PutObject allows, since the request now has to survive a proxy hop and an Envoy timeout. For a claim check, where the whole reason you’re not putting the payload inline the message is a couple megabytes past a broker’s cap, that ceiling isn’t the constraint that bites first.

Building ingest

ingest is deliberately small. It mints the address the way the producer used to mint one for itself, decides which bucket the tenant’s objects live in, and performs the same conditional write:

const s3 = new S3Client({
  region: process.env.AWS_REGION ?? "us-east-1",
  endpoint: process.env.S3_ENDPOINT,
  forcePathStyle: true,
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID ?? "",
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY ?? "",
  },
});

// tenant -> bucket. Empty by default, meaning everyone lands in the shared
// bucket. A tenant gets its own row here the day its objects move, and that
// row is the entire migration.
const TENANT_BUCKETS = JSON.parse(process.env.TENANT_BUCKETS_JSON ?? "{}");

function bucketFor(tenant: string): string {
  return TENANT_BUCKETS[tenant] ?? DEFAULT_BUCKET;
}

and on the request itself:

const tenant = header(req, "x-tenant"); // set by ext_authz from the caller's own token, never the caller
const payload = await readBody(req);

const id = mintObjectId();
const bucket = bucketFor(tenant);
const key = storageKey(tenant, id);
const digest = sha256(payload);

await s3.send(
  new PutObjectCommand({
    Bucket: bucket,
    Key: key,
    Body: payload,
    ContentType: contentType,
    IfNoneMatch: "*", // write-once, enforced by the store itself
    Metadata: { tenant },
  }),
);

const urn = buildObjectUrn({ tenant, id, digest });
sendJson(res, 201, { urn, digest, size: payload.byteLength });

IfNoneMatch: "*" is doing real work there. It’s S3’s native conditional write, added in 2024, and it’s the assertion the whole caching story on the read side depends on: an address is used once, so the object behind it never changes, so the read gateway’s cache can be held for hours without an invalidation strategy. Moving that call from the producer into ingest doesn’t change what it asserts, it just moves who’s responsible for asserting it correctly to one place instead of every writer.

The route table gained one entry to reach it. Envoy matches POST /objects on an exact path, deliberately narrower than the read regex, because there’s no tenant or address to vary yet:

- name: write_object
  match:
    path: "/objects"
    headers:
      - name: ":method"
        string_match: { exact: "POST" }
  route:
    cluster: object_ingest
    timeout: 30s

Everything else in the chain, header_mutation stripping caller-supplied identity headers before anything downstream can trust them, ext_authz checking the token and copying x-subject, x-scope, and x-tenant onto the request, is identical machinery to the read path. There’s no tenant check against the path here, because a write’s path carries no tenant to check against; the token’s claim is the only tenant ingest will ever see, which is also why a caller has no field to lie in even if it wanted to.

The IAM policy ingest holds is intentionally lopsided. It can write, and it explicitly cannot read back or list what it wrote:

{
  "Sid": "WriteObjectsUnderAnyTenantPrefix",
  "Effect": "Allow",
  "Action": ["s3:PutObject"],
  "Resource": ["arn:aws:s3:::demo-objects/objects/*"]
},
{
  "Sid": "NoReadBack",
  "Effect": "Deny",
  "Action": ["s3:GetObject"],
  "Resource": ["arn:aws:s3:::demo-objects/*"]
}

A stolen write credential that can’t read what it wrote is a meaningfully smaller incident. It’s the same least-privilege instinct as the read gateway’s policy, applied to the opposite direction.

What The Producer Actually Lost

The producer’s code barely changed shape. It used to build a key, hash the body, and PutObject directly; now it calls one method on the same GatewayClient a reader already used:

async put(body: Uint8Array, opts: { contentType: string }): Promise<StoredReceipt> {
  const digest = sha256(body);
  const res = await fetch(`${this.#baseUrl}/objects`, {
    method: "POST",
    headers: { authorization: await this.#getAuthHeader(), "content-type": opts.contentType },
    body,
    redirect: "manual",
  });
  if (res.status !== 201) throw new GatewayError(`write failed: ${await excerpt(res)}`, res.status);

  const receipt = await res.json();
  if (receipt.digest !== digest) throw new DigestMismatchError(digest, receipt.digest);
  return receipt;
}

That last check is the write-side mirror of what a reader already does: the gateway computed its own digest from the bytes it actually received, and the client compares that against the one it computed before sending. Corruption in transit gets caught immediately, on the write, instead of days later when some consumer fails to verify it.

What the producer no longer has is an S3 client, a bucket name, a region, or a network route to any of it. Scenario 3 in the demo used to be a claim about the consumer alone; now it checks both:

3. Neither application service can reach the object store at all
  ✔ the bucket-facing network holds only gateway components and the store: object-gateway, ingest, localstack
  ✔ producer holds no AWS credentials
  ✔ producer knows no bucket name, region or endpoint
  ✔ producer is not attached to the bucket-facing network
  ✔ producer cannot even resolve the object store's hostname: localstack: ENOTFOUND
  ✔ consumer holds no AWS credentials
  ✔ consumer knows no bucket name, region or endpoint
  ✔ consumer is not attached to the bucket-facing network
  ✔ consumer cannot even resolve the object store's hostname: localstack: ENOTFOUND

And the write route gets the same two guarantees the read route already had: one recognized path shape, and a scope reading doesn’t confer:

6. The write route serves exactly one path shape, and needs its own scope
  ✔ a shape the write route doesn't recognise → 404: HTTP 404
  ✔ PUT on the write path → 404: this is not the shape it serves: HTTP 404
  ✔ a valid token holding only objects:read is refused: HTTP 403 objects:write
  ✔ the correctly scoped, correctly shaped request succeeds: HTTP 201
  ✔ and it comes back with a reference minted under the caller's tenant

The Part That Was Actually The Point: Moving a Bucket

The last post’s “bonus” section was about the route table as a migration lever on reads: point a cluster at a different backend, and no consumer notices. The obvious follow-up question is whether that same lever exists on the write side, and it does, because the write side is now behind a gateway too.

I moved one tenant, globex, into its own bucket. The change touched three things, and none of them are producer.ts or consumer.ts:

• ingest gained TENANT_BUCKETS_JSON, a map it consults before the PutObjectCommand. A tenant absent from the map falls back to the shared bucket, so this cost one entry, not a migration script.
• Envoy gained a route for GET /objects/globex/{id}, matched ahead of the general read route, pointing at a second nginx-s3-gateway instance configured against the new bucket.
• The bootstrap script gained a second bucket and a second read-gateway identity, scoped to it alone, with the same deny-list on ListBucket and GetObject that the write identity already has.

That’s it. The producer’s request is still POST /objects with a bearer token, and it has no bucket-shaped field to leave stale. The consumer’s request is still GET /objects/{tenant}/{id}, and the tenant it names was always the routing key, not an implementation detail leaking through. Neither call site had anywhere to put an assumption about where the bytes physically live, so there was nothing to go update.

Here’s the scenario proving it, run right after the tenancy check so globex is already established as a distinct tenant by the time it runs:

9. Splitting a tenant into its own bucket needs no producer or consumer code change
  ✔ the write succeeds through the same route acme uses: HTTP 201
  ✔ and mints an address under the globex tenant: urn:demo:object:v1:globex:...
  ✔ the bytes actually landed in globex's own bucket
  ✔ and not in the shared bucket acme still uses
  ✔ and reads back through the gateway exactly like any other object would: HTTP 200
  ✔ acme's existing objects are unaffected by another tenant's migration: HTTP 200

This is the same argument as the Strangler Fig pattern applied at a smaller scale: a migration that happens behind a stable interface, one route at a time, instead of a coordinated rollout across every caller. It’s also the practical version of the choice AWS’s SaaS tenant isolation guidance describes as pool versus silo storage: most tenants pooled in a shared bucket, one tenant siloed into its own because it needs stronger isolation, a different retention policy, or its own compliance boundary. The point of the gateway isn’t picking one model up front, it’s that you can change your mind about which tenants get which model without anyone downstream noticing which one they’re getting.

What’s Still Missing

I want to be as upfront about the gaps here as the last post was about the read side’s costs.

Idempotent retries are still lost. ingest mints a fresh address on every request with no memory of the last one, so a writer that retries after a timeout still gets a second object and an orphaned first one. Fixing this means durable idempotency behind an interface, keyed on something like the Idempotency-Key header, and it’s only possible at all now that a third party other than the writer is minting addresses.

There’s no retention or delete. Every object written here lives forever, and there’s no route to remove one. Both need a scope, an IAM statement, and a lifecycle policy that none of this demo builds yet.

Large payloads still hit A’s ceiling. Anything that needs to stream past what a proxied request comfortably survives needs option C, the two-phase signed upload. When that gets built, I’d also want the gateway to declare its own digest as an RFC 9530 Content-Digest header rather than inventing a header of my own, since “we adopted the standard” travels a lot better between teams than “we made up a header.”

None of these change the shape of what’s built. They’re the next three commits, not a redesign.

When This Doesn’t Fit

Everything above is real work for a real cost, and it isn’t the right amount of ceremony for every writer.

One writer, one team, no plans for a second, is still a fine reason to write directly. The argument in this post is about what happens once a second team starts writing claim checks, or once you need to answer “can we delete this” for a customer, or once a bucket migration is on the roadmap. If none of those are true yet, ingest is a service you’re operating for problems you don’t have.

If your payloads are large and your writers are few, option C is probably the better fit than A. Proxying multi-gigabyte uploads through a gateway process for the sake of isolation you may not need yet is a bad trade. The two-phase signed upload keeps most of the story, drops the size ceiling, and costs an extra round trip instead of a hard proxy limit.

If you don’t have multiple tenants, or don’t expect to move one, the bucket-split payoff doesn’t apply to you. It’s real, but it’s the kind of flexibility that pays for itself the first time you need it and looks like unnecessary indirection every day before that. Don’t build TENANT_BUCKETS_JSON for a single-tenant system.

And if the write path is a background job with no second consumer waiting on it, the whole gateway is probably more machinery than the problem deserves. A nightly batch loader writing to its own bucket, read by nothing else, is the same shape as the “simple application” case from the last post’s read-side conclusion. This pattern is for the moment a system stops being simple, not a default you reach for on day one.

See It In Action

The demo from the last post now runs twelve scenarios instead of ten, covering both directions and the migration:

git clone https://github.com/jamescarr/object-gateway-demo
cd object-gateway-demo
pnpm install
just up      # build and start everything, wait until it is healthy
just demo    # run the scenarios

Two tasks are worth running while the stack is up: just watch-gateway follows Envoy and the authorizer deciding both reads and writes live, and just network-report prints exactly which containers can reach the bucket network, which is now nobody except ingest, object-gateway, object-gateway-globex, and the store itself.

The design notes for the write side, written before ingest existed and updated rather than rewritten once it did, are in docs/write-path.md: the full case for options A, B, and C, the contract a conformance suite would need to enforce across languages, and the exact order the remaining pieces, idempotency, retention, delete, get built in. docs/architecture.md has a write walked end to end the same way the read path was walked in the last post, plus the full failure-mode table for both directions now.

The commit history tells this in order if you’d rather read it that way than take my word for it: giving the writer a gateway, proving the producer actually lost its access, then the bucket split and the scenario that proves it. Same repo, same claim as before, extended to the direction I said I’d get back to.

You can clone it and explore it at your leisure at jamescarr/object-gateway-demo.