fix: use companion metadata object for S3 instead of self-copy#331
Merged
Conversation
Replace server-side CopyObject (copy-to-self) for updating headers and refreshing expiry with a small companion .meta object. S3's CopyObject has a 5GB limit, which caused failures for large objects: The specified copy source is larger than the maximum allowable size for a copy source: 5368709120 The companion stores mutable metadata (ETag, expiry) as a few-hundred-byte JSON object. Immutable headers remain in the data object's user metadata for backwards compatibility — existing objects without companions continue to work via legacy fallback. On read, both are read and the companion is overlaid on top of the base metadata. Expiry refresh becomes a tiny PutObject instead of copying potentially multi-GB data server-side.
alecthomas
requested review from
jrobotham-square
and removed request for
a team
alecthomas
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replace server-side CopyObject (copy-to-self) for updating headers and
refreshing expiry with a small companion .meta object. S3's CopyObject
has a 5GB limit, which caused failures for large objects:
The specified copy source is larger than the maximum allowable size
for a copy source: 5368709120
The companion stores mutable metadata (ETag, expiry) as a few-hundred-byte
JSON object. Immutable headers remain in the data object's user metadata
for backwards compatibility — existing objects without companions continue
to work via legacy fallback.
On read, both are read and the companion is overlaid on top of the base
metadata. Expiry refresh becomes a tiny PutObject instead of copying
potentially multi-GB data server-side.