CDNs and Object Storage: Serving Files, Images and Video at Scale

Pixly stores about 4.6 TB of new photos every day and, at peak, users download about 28 Gbps of images (article 07). Neither number fits the tools we have so far. A relational database is the wrong place for billions of multi-megabyte files. Application servers are the wrong place to stream gigabits of images, and a single region is the wrong place to serve a user on the other side of the world.

Two building blocks solve this. Object storage holds enormous numbers of files cheaply and durably. A content delivery network (CDN) caches those files close to users and serves them without touching your servers. Together they form the "large file" path of almost every modern system, separate from the API path. This article covers both, then puts them together in a small file-storage design.

• The difference between block, file and object storage.
• How object storage works: buckets, keys, metadata, durability, consistency, lifecycle and cost.
• Why clients should upload directly to object storage, using presigned URLs, and how multipart upload works.
• What a CDN does, step by step, and how cache keys, TTLs and headers control it.
• Invalidation versus versioned URLs.
• Serving private content, and what CDNs do for dynamic content.
• A compact design for file storage and sharing, and how these components fail.

You create a bucket (a container) and store objects in it. Each object is a blob of bytes plus metadata (content type, custom tags) and is identified by a key, a string such as photos/u42/p9001/original.jpg. Amazon's documentation describes S3 exactly this way: an object is a file and any metadata that describes it, a bucket is a container for objects, and each object has a key that uniquely identifies it within the bucket (What is Amazon S3?). Google Cloud Storage, Azure Blob Storage and self-hosted systems such as Ceph follow the same model.

Keys can contain slashes, and tools display them like folders, but the namespace is flat. Do not treat listing a "folder" as a query mechanism: keep metadata about files in your database and use object storage for the bytes.

• Scale. You do not provision capacity; you store as much as you need.
• Durability. Data is stored redundantly across devices and facilities. For S3 Standard, Amazon states a design target of 99.999999999% durability and 99.99% availability over a year, with objects stored across at least three Availability Zones (Data protection in Amazon S3).
• Consistency. S3 documents strong read-after-write consistency for PUT and DELETE of objects: once a write succeeds, later reads and listings see it (Amazon S3 data consistency model). Check the documentation of whichever store you use; not every system has always offered this.
• Whole-object writes. You replace an object; you do not edit byte 4,000 in place. That suits files, not records that change constantly.
• Higher latency than local disk. Each request is an HTTP call, often tens of milliseconds. Fine for downloading a photo; wrong for a hot database row.
• Private by default. S3 buckets and objects are private unless you grant access (What is Amazon S3?). Accidentally public buckets are a well-known cause of data leaks.
• Storage classes and lifecycle rules. Rarely read data can move automatically to cheaper, slower classes or be deleted after a period.
• Versioning. Keep previous versions of objects to recover from accidental overwrites and deletions.
• Cost has three parts: storage per GB-month, requests, and data transferred out. For media systems, transfer out is often the largest, which is one more reason for a CDN.

A common pattern:

Database row (photos table)                Object storagephoto_id   = 9001                          photos/u42/p9001/original.jpg  (2 MB)owner_id   = 42                            photos/u42/p9001/w1080.jpg     (200 KB)caption    = "Sunset"                      photos/u42/p9001/w320.jpg      (30 KB)status     = visibleobject_key = photos/u42/p9001/created_at = 2026-10-04 18:22

The database answers questions ("Asha's photos, newest first", "is this photo visible to me?"). Object storage holds bytes. Each does what it is good at.

The naive design sends the file to an API server, which writes it to object storage. That ties up API server connections, memory and bandwidth for the whole upload, which on a slow mobile network can take minutes. With Pixly's 70 uploads per second at peak, each lasting tens of seconds, hundreds of API connections would be busy just relaying bytes.

The better design lets the client upload directly to object storage, using a presigned URL: a URL that the API server creates and signs with its credentials, granting permission to perform one specific operation on one specific key, for a limited time. Amazon documents presigned URLs for exactly this purpose (Download and upload objects with presigned URLs), and Microsoft describes the general idea as the Valet Key pattern: give clients a restricted, time-limited token for direct access to a resource (Valet Key pattern).

1. App  --POST /v1/photos {caption, size, type}-->  API server2. API server: validate, create photo row (status = uploading),              sign an upload URL for key photos/u42/p9001/original.jpg,              valid for 10 minutes, for this content type3. App  <--201 {photoId, uploadUrl}--  API server4. App  --PUT bytes-->  Object storage (directly, using the signed URL)5. Object storage --"object created" event--> Queue --> Workers6. Workers: verify the file, create thumbnails, set status = visible

API servers now handle only small JSON requests. Object storage, built for this, handles the bytes.

Security rules for presigned uploads:

• Short expiry (minutes).
• Fix the key on the server side; never let the client choose an arbitrary key.
• Restrict content type and maximum size where the store supports it, and verify after upload anyway: check the real file type and size, scan it, and only then mark it visible.
• Upload into a private "incoming" area; promote the file only after checks pass.

A 2 GB video upload over a mobile network will be interrupted at some point. Restarting from zero each time is painful. Multipart upload splits a file into parts that are uploaded independently, in parallel if you like, and then assembled. If one part fails, only that part is retried.

Amazon's S3 documentation describes the process (initiate, upload parts, complete), recommends multipart upload for objects of 100 MB or larger, and lists the advantages: improved throughput through parallel parts, quick recovery from network issues, and the ability to pause and resume (Uploading objects using multipart upload). It also recommends a lifecycle rule to delete incomplete multipart uploads, because abandoned parts are otherwise stored and billed indefinitely.

Client                         API server                    Object storage  | start upload (size, type) ---->|-- initiate multipart ------>|  |<-- uploadId + signed URLs for parts 1..N --|                  |  |-- PUT part 1 ------------------------------------------------>|  |-- PUT part 2 ------------------------------------------------>|  (parallel, retry per part)  |-- ...                                                         |  |-- complete (part list) ------->|-- complete multipart ------->|

Article 40 builds video upload on exactly this pattern.

A CDN is a network of caching servers, often called edge locations or points of presence, spread across many cities. Each edge is a reverse proxy with a cache (article 08). DNS or anycast routing sends each user to a nearby edge. Amazon describes CloudFront as a web service that speeds up distribution of static and dynamic content by delivering it through a worldwide network of edge locations, routing each request to the location with the lowest latency (What is Amazon CloudFront?). Other CDNs (Google Cloud CDN, Azure Front Door, Cloudflare, Akamai, Fastly and more) work on the same principles.

Your own servers or bucket, where the CDN fetches content it does not have, are called the origin.

User in Delhi   |  DNS sends them to the nearest edge   vEdge location (Delhi)   |-- cache HIT?  --> return the file immediately (a few ms away from the user)   |-- cache MISS  --> fetch from the origin (perhaps via a regional mid-tier cache),   |                   store it according to its cache rules, then return it   vOrigin: object storage bucket or your web servers (for example in Mumbai)

The next user in Delhi who asks for the same file gets a hit. For popular content, the origin sees only a small fraction of requests.

Benefits, in order of how often they matter:

1. Lower latency for distant users: the round trips from article 03 are to a nearby edge, not to your region.
2. Less load and egress at the origin: Pixly's 28 Gbps became about 2.8 Gbps at a 90% hit ratio (article 07).
3. Absorbing traffic spikes and attacks: a CDN spreads load across a very large network and can absorb floods that would overwhelm one data centre.
4. Edge TLS termination and connection reuse, even for content that is not cached.

CDNs follow the HTTP caching rules (RFC 9111), mostly controlled by the origin's response headers. MDN's guide to HTTP caching explains these in detail (HTTP caching). The ones you need:

CDN providers also let you set default and maximum TTLs per path, which override or complement these headers.

The cache key decides which requests count as "the same". By default it is usually the host and path, sometimes with the query string. Two common mistakes:

• Too much in the key: including every query parameter (?utm_source=...) or every header splits one file into thousands of cache entries, and the hit ratio collapses.
• Too little in the key: if a response varies by language, device or (worst) by logged-in user, but the key ignores that, the CDN serves one user's version to everyone. Serving a personalised page from a shared cache is one of the classic ways private data leaks.

Rule: cache only responses that are identical for everyone who shares the key, and include in the key exactly the inputs that change the response.

When a file changes, cached copies at hundreds of edges must stop being served. Two options:

1. Invalidate (purge): ask the CDN to drop cached copies of a path. The next request goes to the origin.
2. Version the URL: put a version or content hash in the URL (/assets/app.3f9a1c.js, /img/p9001/w1080-v2.jpg). A changed file gets a new URL; the old URL can be cached forever because its content never changes.

Amazon's CloudFront documentation recommends primarily using versioned file names when you update files frequently, because versioning works even when users have the old version cached in their browser or a corporate proxy, makes rolling forward and back easy, and avoids invalidation charges (Invalidate files).

Use versioned URLs with long TTLs as the default for anything you control the links to (images, scripts, stylesheets). Keep invalidation for emergencies (a file that must be removed now) and for URLs you cannot change (for example /robots.txt).

• Static content (images, video segments, scripts, fonts, downloadable files) is identical for everyone: ideal for long CDN caching.
• Semi-dynamic content (a public product page, a news article, a live score) can be cached for seconds to minutes, which still removes most origin traffic (article 05's live-score example).
• Dynamic, personalised content (your feed, your cart, your account) is generally not cached at a shared edge. Routing it through the CDN still helps: TLS is terminated nearby, the edge keeps warm connections to your origin over a good network, and attack traffic is filtered.

Some files are private: a photo from a private account, a paid course video, an invoice PDF. Options:

• Signed URLs or signed cookies at the CDN: your API issues a URL valid for a short time; the CDN checks the signature before serving. The file is cached at the edge, but only holders of a valid signature can fetch it.
• Restrict the origin so only the CDN can read the bucket, preventing users from bypassing the CDN's checks by going to the bucket directly.
• For highly sensitive data, skip shared caching entirely.

In a pull CDN (the common case), edges fetch from the origin on a miss. In a push model, you upload content to the CDN's storage ahead of time. Pull is simpler and fits most designs; pushing or pre-warming helps for a known big release, such as a new episode that millions will request in the same minute.

The brief asks for a Level 1 "file storage" design. Because it is built almost entirely from this article's components, here is a compact version in the course's method. (Article 40 extends it to video.)

Requirements. Users upload files up to 5 GB, list and download their files, and share a file by link (anyone with the link can download until it expires or is revoked). Out of scope: real-time co-editing, desktop sync, full-text search.

Non-functional. Uploads must survive network interruptions; an acknowledged file is never lost; downloads are fast worldwide; shared links must not expose other files.

Estimate (assumptions). 5 million DAU; 1 million uploads a day at an average of 5 MB = 5 TB a day, about 1.8 PB a year; 10 million downloads a day at 5 MB = 50 TB a day, about 4.6 Gbps on average, roughly 10 Gbps at peak.

API.

Data model.

files(file_id PK, owner_id, name, size_bytes, content_type,      object_key, sha256, status, created_at)        index (owner_id, created_at)shares(share_id PK (random, unguessable), file_id, created_by,       expires_at, revoked_at)

Architecture.

Client --API--> Load balancer --> API servers --> Metadata DB (+ replicas)   |                                  |   |  signed PUT (multipart)          +--> Queue --> Workers (virus scan, checksums)   vObject storage (private bucket) <---- CDN (origin, read-only access) <---- signed GET

Key decisions.

• Bytes never pass through API servers: uploads use presigned multipart URLs, downloads use signed URLs served through the CDN.
• The share link resolves to a share_id row; the API checks expiry and revocation each time and only then issues a short-lived signed download URL. Revoking a link takes effect on the next request; an already-issued download URL stays valid only for its few minutes.
• Deduplication (optional): the stored sha256 lets you detect identical uploads and store the bytes once, at the cost of reference counting on delete.
• Lifecycle rules delete abandoned multipart uploads and move rarely downloaded files to a cheaper storage class.

Failures. Upload interrupted: the client retries the failed parts. The API server crashes after creating the row but before the client uploads: a cleanup job deletes rows stuck in uploading for over a day. The CDN has a problem: downloads can fall back to signed URLs served directly from the bucket, at higher latency and cost. A region-wide storage outage is the hardest case; cross-region replication of the bucket is the answer if requirements justify the cost (article 24).

• Storing images or videos as database blobs.
• Uploading large files through API servers.
• Letting clients choose object keys, or making buckets public to "make images work".
• Caching personalised API responses at the CDN.
• Cache keys that include tracking parameters, destroying the hit ratio.
• Relying on purges instead of versioned URLs for frequently changed assets.
• Forgetting lifecycle rules for incomplete uploads and old data.

• The moment a design has user-generated media, say: "files go to object storage; the database stores metadata and the object key; clients upload directly with presigned URLs; downloads go through a CDN."
• Quote the egress estimate to justify the CDN.
• For private media, mention signed URLs with short expiry.
• For large files, mention multipart upload and resumability.

• Next: Message Queues, Pub/Sub and Event Streams: How Asynchronous Systems Work, which powers the "after upload" processing in this article.
• Later: the URL shortener's Pastebin variant (article 30) stores pastes in object storage; video streaming (article 40) builds a full upload, transcoding and CDN pipeline.