Amazon S3 Multipart Uploads
Amazon S3 multipart upload lets you create one object by uploading it as a set of independently transferred parts, then asking S3 to assemble those parts. The parts can be uploaded concurrently and retried separately, while S3 exposes the result as one ordinary object.
The problem multipart upload solves
A single request for a large object is fragile. It may take a long time, consume a large buffer, hit a connection timeout, or fail near the end after all the earlier bytes have already crossed the network. Retrying the whole request wastes that work.
Multipart upload divides the object into chunks. A failed transfer then requires only one part to be sent again. Several parts can also travel in parallel, which improves throughput for large files.
S3 places limits on this process: an upload can contain at most 10,000 parts, each part is generally at least 5 MiB except for the final part, and the completed object can be up to 5 TiB. The client chooses the part size, so it must choose a size that stays within the part-count limit.
The three-stage protocol
A multipart upload has three distinct stages:
- Initiate the upload. S3 returns an upload ID, an identifier that distinguishes this unfinished upload from every other upload for the same object key.
- Upload parts using that upload ID. Each request supplies a part number from 1 through 10,000 and the bytes for that part.
- Complete the upload by sending S3 the list of successfully uploaded part numbers and their returned ETags. S3 then assembles the object in ascending part-number order.
The object key does not become the completed object during the first two stages. S3 stores the uploaded parts as components associated with the upload ID. A reader fetching the key does not see a half-built object.
Part numbers describe order, not necessarily request order. Part 7 can arrive before part 2, and requests can run concurrently. If the client uploads another part with the same number, the newer upload replaces the earlier version of that part for this multipart upload. This makes an individual part straightforward to retry.
The completion request is the commit point. It names the upload ID and the parts that should be used. S3 checks that the referenced parts exist, then constructs one object. A missing part, an invalid ETag, or an invalid part sequence causes completion to fail rather than silently producing a damaged object.
The client must retain the upload ID and the per-part results. Losing either makes reliable completion difficult, even if S3 has already received all the bytes.
How each part is verified
A part needs two kinds of correctness:
- Identity: the server must associate the bytes with the intended upload ID and part number.
- Integrity: the bytes received must be the bytes the client intended to send.
S3 uses HTTPS for transport, but transport encryption does not replace an end-to-end content check. A client can provide a checksum with an upload-part request. The checksum is a compact digest calculated from the part's bytes using an algorithm such as CRC-32C, SHA-256, or another algorithm supported by the relevant S3 API. S3 calculates the same checksum from the bytes it receives. If the values differ, S3 rejects the part instead of storing it as successful. It returns checksum information when the part is accepted, which the client can record.
Older S3 integrations commonly use the Content-MD5 header for this purpose. MD5 here is being used as an accidental-corruption check, not as a security design for passwords or signatures. Newer clients may use S3's checksum headers and negotiate a checksum algorithm when initiating the upload.
S3 also returns an ETag for each successful part. The completion request includes those ETags so S3 can identify the exact uploaded version of each numbered part. An ETag is an opaque response value for multipart-upload purposes; do not assume it is always the MD5 hash of the part. In particular, a completed multipart object's ETag is often a multipart-specific value rather than the MD5 of the entire object, and encryption settings can change ETag behavior.
If an application needs to verify the completed file, it should use an explicitly managed checksum or an S3 checksum feature rather than treating the ETag as a universal file hash. Part checksums protect each transfer; completion combines the accepted parts in the declared order.
Assembly and visibility
Suppose a client splits a file into parts 1, 2, and 3 and uploads them in the order 2, 1, 3. The completion request still lists 1, 2, 3. S3 reads those stored parts in that order and creates the final object. The resulting object has the normal S3 behavior for reads, metadata, permissions, versioning, and lifecycle policies.
This is why multipart upload is not the same as independently writing several objects and joining them in application code. S3 owns the temporary components and performs the final assembly as one object operation. Applications do not need to download the parts, concatenate them, and upload the result again.
SDKs often hide this protocol behind a high-level “managed,” “transfer,” or “multipart” upload helper. A log mentioning an upload ID, UploadPart, CompleteMultipartUpload, or AbortMultipartUpload is exposing the underlying mechanism.
Why abandoned uploads need cleanup
An unfinished multipart upload is not a visible object, but its successfully uploaded parts remain stored. They consume S3 storage and can incur storage charges. S3 does not automatically infer that a client has crashed, lost power, or stopped caring about an upload merely because no new part has arrived.
A client that gives up should explicitly abort the multipart upload. Aborting tells S3 to discard the stored parts associated with that upload ID. A successful completion also releases the temporary state as part of producing the final object.
Production buckets should usually have a lifecycle rule that aborts incomplete multipart uploads after a suitable number of days. This is a backstop for crashed clients and forgotten upload IDs, not a substitute for prompt application cleanup. Operators can also list incomplete multipart uploads to find leaks and investigate unexpectedly high storage use.
The practical model is therefore: initiate, upload and verify parts, complete with the recorded part list, and abort—or let a lifecycle rule expire—the upload if completion will not happen.