Live on Product Hunt Intraface launched today. Less steering, more vacations. See the launch
Start for free

What is content-addressable storage?

Content-addressable storage is a way of storing data where the key is a hash of the content itself, not a path or a name you choose. Store the same bytes twice and you get the same key, so duplicates collapse into one copy, and anyone holding the key can check that what they fetched is what they asked for.

Part of Glossary: the terms behind MCP, coding agents and shared pages, defined plainly.

How it works

Three files are hashed; the two with identical bytes get the same key and are stored once, the third gets its own key.logo.pngversion 3logo.pngversion 4, same byteshero.jpgversion 4hash the bytesSHA-1, SHA-256the storekeyed by hashsame bytes, same key:stored oncedifferent bytes:new key, new object1. hash1. hash1. hash2. key 3f9a, stored once2. key b71c, new object
  1. Take the bytes of the file and run them through a cryptographic hash such as SHA-1 or SHA-256. The result is the key.

  2. Store the bytes under that key. Git puts each object in .git/objects under the first two characters of its hash.

  3. Ask for the same bytes again and the hash matches an existing key, so nothing new is written. Change one byte and you get a new key and a new object.

  4. To read, fetch by key and hash what came back. If it matches, the content is intact and untampered, wherever it came from.

  5. Names and paths live one level up: a tree, a manifest or a version points at keys, so renaming a file does not copy its bytes.

Source: Pro Git: Git Internals, Git Objects

Content addressing versus location addressing

Most storage is addressed by location. A path like /images/logo.png or a URL says where to look, and whatever is there now is what you get. Rename the file and the address breaks; overwrite it and the old content is gone; keep two copies and nothing knows they are the same. Content addressing turns this round. The address is computed from the bytes, so it says what you will get, not where. The same bytes always produce the same key, different bytes always produce a different one, and anyone who has the key can verify what they fetched by hashing it again.

That gives you three things for free: deduplication, because identical content maps to one key; integrity, because a mismatch between key and content is detectable; and immutability, because you cannot change the content behind a key without changing the key.

Example: Git's object store

The Pro Git book opens its internals chapter by calling Git a content-addressable filesystem, and underneath, a simple key-value data store. Give Git some content and it hands back a 40-character SHA-1 key. The hash covers a short header (the object type, the size in bytes, a null byte) followed by the content, and the compressed result is written to .git/objects, in a directory named by the first two characters of the key with the remaining 38 as the filename.

A blob is just content with no filename. Filenames live in tree objects, which list a mode, a type, a key and a name for each entry, and a commit points at a tree and its parent commits. So a file that is unchanged across a hundred commits is one blob, referenced a hundred times. Renaming it changes a tree, not the blob. That is why Git history is cheap and why the same content shows up as the same hash in every clone in the world.

IPFS applies the same idea across machines. Its content identifier, or CID, is derived from the content, with SHA-256 by default and the algorithm recorded inside the identifier. Adding the same content on two nodes with the same settings gives the same CID. One subtlety from the IPFS docs: large content is split into blocks arranged in a graph, and the CID hashes the root block, so it will not equal the file's plain SHA-256 checksum.

Where it shows up

Any system that keeps many versions of similar things wants content addressing: package caches, build caches, container image layers, backup tools, and version histories of documents. A shared page that agents publish new versions of is a good example: most publishes change one file and leave the images alone, and with content addressing the unchanged files are stored once. If your system keeps every version, as an audit trail should, this is what stops storage growing with every publish.

Common mistakes

  • Treating the hash as a name. Keys are for machines. Keep a layer of names, trees or manifests that point at keys, and let humans use those.
  • Assuming the hash equals a checksum of the file. Git hashes a header plus content; IPFS hashes a root block. Compare like with like.
  • Deleting by key. Many references may share one object. Garbage collect from the roots instead of deleting what looks unused.
  • Picking a weak hash. The whole scheme rests on collisions being impractical. Use a current cryptographic hash and let the format record which one.

FAQ

Is content-addressable storage the same as deduplication?
Deduplication is a result of it. Because the key is a hash of the bytes, storing identical content twice produces the same key and one copy. Content addressing is the scheme; deduplication, integrity checks and immutability follow from it.
Which hash does content-addressable storage use?
Whatever the system chooses, as long as it is cryptographic. Git's object store, as described in Pro Git, uses SHA-1 over a header plus the content. IPFS uses SHA-256 by default and records the algorithm inside the CID so it can change later.
Why does an IPFS CID differ from a file's SHA-256 checksum?
IPFS splits content into blocks arranged in a graph and the CID hashes the root block, not the raw file. Only a file stored as a single raw block has a CID whose hash matches its plain checksum.

Stop sending HTMLover Slack.

Create an account in a minute, connect your coding agent, and make your first board.