Git Packfiles and Delta Compression
A Git packfile is a file that stores many Git objects together, often storing one object as a compact difference from another. Delta compression records only how to transform a similar base object into the target object, reducing repeated data across commits.
Packfiles solve two related problems. First, Git repositories can contain millions of objects, and storing each one as a separate file creates filesystem overhead and makes copying the repository inefficient. Second, successive versions of a source file are usually similar, so compressing each version independently misses an opportunity to store their shared content only once.
The objects being packed
Git’s object database contains immutable, content-addressed objects: the object’s identifier is a hash of its type, length and contents. Depending on the repository, that hash uses SHA-1 or SHA-256. The main object types are:
- A blob, which contains file contents.
- A tree, which records a directory’s entries and their names.
- A commit, which records a snapshot’s top-level tree, parent commits and metadata.
- An annotated tag, which names another object with additional metadata.
In an unpacked repository, each object is normally stored in its own compressed file under the repository’s object directory. The filename is derived from the object’s hash. This makes direct lookup straightforward, but a large project accumulates many small files. It also stores similar blobs separately: every revision of a file has its own complete compressed content.
A packfile changes the physical representation without changing Git’s logical object database. The objects still have the same identifiers and contents; they are simply stored together.
What a packfile contains
A packfile begins with a header identifying the format, its version, and the number of packed objects. After that comes a sequence of packed object entries, followed by a checksum for integrity.
Each entry describes an object’s type and its uncompressed size. The object data is then compressed, usually with zlib. For an ordinary entry, that compressed data is the complete object content, so Git can reconstruct the usual object header and contents directly.
A packfile does not need to store the object’s full hash beside every entry. The accompanying pack index, usually a file ending in .idx, maps object identifiers to locations in the packfile. Git can therefore find an object by hash, use the index to locate its entry, decompress it, and verify or reconstruct its content.
The
.packfile holds the data; the.idxfile makes finding an object in that data practical.
The index also contains enough information to enumerate the pack’s object identifiers and to check whether a requested object belongs to that pack. A repository may have several packfiles, each with its own index, although newer Git features can provide an additional index across multiple packs.
How delta compression works
For similar objects, Git can store a delta instead of complete contents. A delta has two sizes: the size of its base object and the size of the reconstructed result. Its instructions then describe how to create the result from the base.
There are two instruction kinds:
- Copy a range of bytes from the base, given an offset and length.
- Insert a short sequence of new bytes that does not occur in the copied material.
Suppose two versions of a source file differ by one line. The delta can copy most of the old file and insert the changed line. The result is logically identical to storing the complete new blob, but usually requires much less space.
A delta entry identifies its base in one of two ways. An offset delta stores a backward offset to another entry in the same packfile. A reference delta stores the base object’s hash, allowing the base to be in another packfile or elsewhere in the object database. The pack format records which kind is being used, so Git knows how to locate the base.
Delta chains
A delta’s base may itself be a delta. This creates a delta chain:
complete blob A
-> delta producing blob B
-> delta producing blob C
To reconstruct C, Git first reconstructs B from A, then applies the delta for C. Chains save more space because many versions can share a carefully selected base, but they add work during reads. A long or poorly chosen chain means more decompression and more delta application before Git can return the requested object.
Git’s packing algorithms therefore balance size against access cost. The exact choice depends on similarity, available memory, configured limits and the repacking operation. A pack does not change an object’s hash: after applying all required deltas, Git obtains the same canonical object bytes and can calculate the same identifier.
What repacking changes
Repacking is the process of reorganizing objects into new packfiles. It can collect loose objects, combine existing packs, choose new delta bases, and write new index files. The result may have fewer packfiles, fewer filesystem entries and a smaller total size.
A typical repository becomes smaller because repacking can:
- Remove repeated storage of similar object contents through deltas.
- Replace many loose compressed files with sequential pack data.
- Consolidate duplicate physical representations of objects.
- Choose better bases and delta chains across objects that were created at different times.
Repacking also changes how lookup proceeds physically. Git still searches by the object’s hash, but it may now find that hash in a pack index instead of a loose-object path. The index gives the pack offset, after which Git reads and reconstructs the entry. If the entry is a delta, Git follows its base reference or offset and applies the chain.
The object identifier and the visible history do not change merely because an object moved into a new pack or received a different delta representation. Pack offsets, compression choices and chain layouts are implementation details and can change on the next repack.
Old packs cannot be deleted while a Git process may still need them, so repacking writes replacement files and removes obsolete ones only when it is safe. Unreachable objects—objects no longer reachable from refs such as branches or tags—may remain for a while as recovery data; later pruning can remove them.
This is why an incident mentioning .pack, .idx, “unable to find base,” or expensive delta resolution is describing Git’s storage layer, not a different kind of commit. The repository still contains the same content-addressed objects; packfiles determine how economically and how quickly those objects are stored and reconstructed.