Object Storage File Delivery and Large-Data Workflows

Published by Programming Line in System Design and Distributed Systems. Last updated .

Purpose and outcome

Object storage is the usual backbone for delivering files and moving large data because it separates byte storage from application servers. A web service can create a report, video, backup, model artifact, or data export, place it in a bucket under a key, and let storage and edge infrastructure handle upload, download, retention, and replication. The design outcome is simple: application nodes stop pretending to be file servers, while clients still receive durable, addressable bytes with predictable authorization and transfer behavior.

In system design, this lesson sits at the point where distributed coordination meets scale. The coordination problem is not consensus between every downloader. It is coordinating object names, metadata, upload completion, event processing, cache invalidation, and temporary access so many independent workers and clients can act on the same large file without corrupting it or overloading one service.

How object storage works internally

An object store organizes data as buckets, object keys, object bytes, and metadata. A key such as tenant-42/exports/orders-2026-09.csv is not a directory path in the filesystem sense. It is an ordered string in a flat namespace. Providers may display slashes as folders, but the storage system stores and indexes keys. This distinction matters when designing prefixes for listing, lifecycle rules, event subscriptions, and access policies.

Writes normally create a whole object version. Instead of modifying block 900 inside a file, a client uploads a new object or a new version of the same key. Large uploads are split into numbered parts. The client initiates a multipart upload, sends parts independently, then commits a manifest that says which part numbers form the final object. Until completion, readers should not see a partial object. This is why object storage fits immutable artifacts, exports, logs, media, and lakehouse-style data files better than in-place database pages.

Object metadata is stored beside the bytes. Important fields include content type, content length, entity tag or checksum, last modified time, server-side encryption mode, storage class, custom metadata, and cache control. Download clients use these fields to resume transfers, validate integrity, and decide whether a cached copy is still useful. Many systems also maintain application metadata in a relational table: object key, owner, logical state, expected checksum, row count, and deletion policy. The table answers business questions; the bucket stores bytes.

Delivery commonly uses a signed URL or signed request. The application checks the user’s authorization, signs a request for a specific bucket, key, method, and expiration, then returns that URL to the client. The client downloads directly from storage or through a CDN. The application is no longer in the data path, so it avoids tying up worker threads and network bandwidth for a multi-gigabyte response. For repeated public or semi-public downloads, a CDN caches objects near users, honors byte-range requests, and shields the bucket from hot traffic.

API anatomy and configuration

The core API verbs are small: put an object, get an object, head an object, delete an object, list objects by prefix, initiate multipart upload, upload part, complete multipart upload, and abort multipart upload. Around those verbs are policies. Bucket policy answers who can do which action. CORS rules answer which browser origins may upload or download. Lifecycle rules move old objects to cheaper classes or expire them. Event rules notify queues or functions when objects are created or deleted.

A delivery design usually names five things explicitly: the bucket, the key scheme, the content metadata, the authorization path, and the post-upload workflow. A good key scheme includes tenant or domain boundaries, time partitioning where listing by date is useful, and an immutable component such as a job id or content hash. Reusing a key for mutable content is possible, but it complicates caches and replayed events.

Example 1: Generate a signed download URL

The first example models the shape of a signed file delivery URL. Real providers add canonical request hashes and cryptographic signatures, but the important design idea is that the application grants narrow, temporary access to one object while keeping the bucket private.

from urllib.parse import urlencode

bucket_host = "https://objects.example.com"
key = "exports/2026-09/orders.csv"
params = {
    "response-content-disposition": "attachment; filename=orders.csv",
    "X-Expires": "900"
}
url = f"{bucket_host}/{key}?{urlencode(params)}"
print(url)

The deterministic output begins with https://objects.example.com/exports/2026-09/orders.csv? and includes an attachment filename plus a 900 second expiration parameter. In production, the signature binds the method, key, headers, and expiry. If a user changes the key or extends the expiration, signature verification fails.

Example 2: Split a large upload into parts

Multipart upload improves reliability for large objects. A failed 8 MiB part can be retried without resending a 20 MiB file, and independent parts can be uploaded in parallel when the client has bandwidth.

import hashlib

chunk_size = 8 * 1024 * 1024
parts = []
for part_number, start in enumerate(range(0, 20 * 1024 * 1024, chunk_size), start=1):
    end = min(start + chunk_size, 20 * 1024 * 1024)
    payload = f"bytes:{start}-{end}".encode()
    checksum = hashlib.sha256(payload).hexdigest()
    parts.append({"part": part_number, "start": start, "end": end, "sha256": checksum[:12]})

for part in parts:
    print(part)

This produces three part descriptors: two full 8 MiB ranges and one final 4 MiB range, each with a short checksum preview. The expected behavior is stable part numbering: part 1 covers bytes 0 through 8388608, part 2 covers 8388608 through 16777216, and part 3 covers the remainder. The complete step must reference the accepted part numbers, otherwise the final object is missing data or the commit is rejected.

Example 3: Process object-created events idempotently

Object-created events are often delivered at least once. A queue, function, or workflow may see the same notification twice, especially around retries. Consumers therefore need an idempotency key based on the object identity they actually processed.

events = [
    {"key": "raw/a.csv", "etag": "e1", "event": "created"},
    {"key": "raw/a.csv", "etag": "e1", "event": "created"},
    {"key": "raw/a.csv", "etag": "e2", "event": "created"},
]
processed = set()
for event in events:
    job_id = f"{event['key']}#{event['etag']}"
    if job_id in processed:
        print(f"skip duplicate {job_id}")
        continue
    processed.add(job_id)
    print(f"transform {job_id} -> curated/{event['key'].split('/')[-1]}")

The output is transform raw/a.csv#e1 -> curated/a.csv, then skip duplicate raw/a.csv#e1, then transform raw/a.csv#e2 -> curated/a.csv. The second event is skipped because it describes the same object version. The third is processed because the entity tag changed, meaning the bytes are different even though the key is the same.

Example 4: Choose a delivery pattern

Small files can still be returned by the application, but large or repeated downloads should move out of the application path. This decision helper captures the common routing logic.

def choose_delivery(size_mb, private, repeated_downloads):
    if size_mb > 100:
        return "multipart upload, object storage, signed URL, CDN range caching"
    if private:
        return "object storage with short-lived signed URL"
    if repeated_downloads:
        return "public object through CDN with immutable cache key"
    return "direct application response is acceptable"

for request in [(25, True, False), (900, True, True), (2, False, False)]:
    print(request, "=>", choose_delivery(*request))

The 25 MB private file uses a short-lived signed URL. The 900 MB file uses multipart upload plus signed CDN delivery with range caching. The 2 MB public, non-repeated response can remain a direct application response because storage offload would add complexity without meaningful benefit.

Design choices and trade-offs

Immutable keys versus overwritten keys. Immutable keys such as exports/job-924/result.csv are easier to cache, audit, and retry. Overwritten keys such as exports/latest.csv are convenient for humans but can produce stale CDN responses and ambiguous event handling. A common compromise is to store immutable objects and update a small database pointer called latest.

Presigned URL versus application proxy. Presigned URLs reduce server load and support browser-to-bucket transfers. Application proxies centralize authorization and logging but make the application carry every byte. Use a proxy when bytes must be transformed inline, scanned synchronously, or hidden from direct storage endpoints. Otherwise, sign narrow access and keep the bucket private.

One huge object versus partitioned objects. A single archive is simple to download. Partitioned objects are better for analytics, partial retries, and parallel processing. Data workflows usually prefer partitioned files by date, tenant, or shard, while user-facing exports often prefer one packaged artifact plus a manifest.

Lifecycle cost versus retrieval speed. Cold storage classes reduce cost for old backups and compliance archives, but restore operations can take time and may cost extra. Do not move files to cold tiers until the product can tolerate delayed retrieval and can show users that restore is pending.

Failure modes and troubleshooting

Symptom: users receive 403 when opening a download link. The cause is usually an expired signature, clock skew during signing, a key mismatch, or a policy that denies the signed principal. Diagnose by checking the URL expiry, server clock, canonical key, and storage access logs. Correct by shortening the signing path to one library, synchronizing clocks, and generating a fresh URL only after the application authorization check passes.

Symptom: downloads start fast and then fail near the end. The likely cause is a client, proxy, or CDN timeout on a large object without range resume. Inspect response headers for Accept-Ranges, compare failure byte positions, and retry with a range request from the last successful offset. Correct by enabling range support through the CDN, avoiding application proxy buffering, and using a download client that resumes.

Symptom: a workflow processes the same file twice. The cause is at-least-once event delivery or a key overwritten between notifications. Check queue delivery counts, object entity tags, and the processing table. Correct by storing an idempotency record for key plus version or checksum before running expensive transforms.

Symptom: listing a prefix is slow or incomplete for a job. The cause may be millions of objects under one logical prefix, pagination not being followed, or a design that uses list operations as a database query. Check continuation tokens and request counts. Correct by recording expected outputs in job metadata and treating bucket listing as reconciliation, not the main source of workflow truth.

Security, performance, and reliability implications

Keep buckets private by default, grant service roles the minimum verbs and prefixes they need, and use short expirations for signed URLs. For browser uploads, constrain CORS to known origins and allowed headers. For sensitive data, use server-side encryption and ensure logs do not leak signed URLs because a bearer URL can be used by anyone until it expires.

Performance depends on parallelism, locality, and caching. Multipart upload uses parallel network streams. CDN delivery keeps hot bytes close to clients and supports range requests for media players and resumable downloads. Checksums protect against corrupted transfer and also make retry decisions objective. Reliability comes from treating uploads as state machines: requested, uploading, completed, verified, published, expired. Each transition should be explicit in application metadata.

Hands-on lab: private export delivery

Prerequisites: a local Python interpreter and a temporary directory. This lab simulates the control plane of a private export without requiring a cloud account.

  1. Create a logical export record with owner id, object key, expected size, and state requested.
  2. Write or imagine the export bytes under an immutable key such as exports/job-1001/orders.csv.
  3. Compute a checksum and update the record to completed only after the checksum matches.
  4. Generate a signed URL with a short expiry for the matching owner.
  5. Verify that changing the key would require a new signature and that the old logical record still points to the original object.

Verification: the completed record should contain exactly one immutable key, one checksum, one owner, and one expiration time for the current download grant. Cleanup: delete the temporary object, remove the export record, and revoke any still-valid test URLs by rotating the signing key or waiting for expiry. In a real bucket, also abort abandoned multipart uploads so incomplete parts do not accumulate cost.

Assessment exercises

  1. A customer asks for a permanent public URL to private invoices. Design a safer flow using object storage and explain how expiration, ownership checks, and audit logging work.
  2. You need to deliver a 4 GB video to mobile clients over unreliable networks. Choose object metadata, CDN behavior, and client retry behavior that make partial download recovery possible.
  3. An analytics job writes 2,000 partition files and a downstream workflow starts too early after seeing the first object-created event. Propose a completion marker or manifest design that prevents partial reads.
  4. A team stores daily reports at reports/latest.csv and users complain about stale downloads. Explain why the CDN behaves this way and redesign the key scheme.
  5. Compare storing 10 TB as one archive with storing it as date-partitioned objects. Discuss download simplicity, processing parallelism, retry cost, and lifecycle policy.

Summary

Object storage file delivery works by making large byte sequences durable, addressable, and independently transferable while the application controls names, metadata, authorization, and workflow state. The strongest designs use immutable keys, multipart transfer for large objects, signed access for private delivery, idempotent event processing, explicit metadata records, and CDN range caching where downloads are hot or resumable. These choices turn large-data movement from a fragile application response into a coordinated storage workflow.