arthrop0d

Brief articles on varied topics.

HTTP/2 HPACK Header Compression

web developmentperformance

HPACK is the header-compression format used by HTTP/2. It represents repeated HTTP header names and values as compact indexes into shared tables, and can additionally encode text with a purpose-built Huffman code.

The problem HPACK solves

HTTP headers are often repetitive. A browser may send the same :authority, cookie, user-agent, and accept values on many requests. A server may repeat content-type, cache-control, or set-cookie patterns in responses. Without compression, each request carries those strings again, even though both endpoints already know them.

HTTP/1.1 could compress the body with gzip, but headers were exchanged separately and usually remained plain text. This matters most for many small requests: the headers can be as large as, or larger than, the useful response. TLS encrypts headers but does not remove their repetition.

HPACK addresses repetition by letting one endpoint say, in effect, “use header field number 17 again” instead of sending the field's text. It also reduces the size of text that must still be sent.

HPACK compresses header fields, not the HTTP message body.

The static table

Every HPACK implementation starts with the same static table: a fixed list of common header fields defined by the HTTP/2 standard. It contains frequently used pseudo-headers, such as :method: GET and :status: 200, as well as common names such as content-type and cache-control.

A header field can be represented by an index into this table. For example, a request method of GET can use the static entry for :method: GET, requiring no transmission of either string.

The table also contains entries for names without a particular value. This allows HPACK to reuse a name while sending a new value. Conceptually, a header might be encoded as:

indexed name: content-type
literal value: application/custom-format

The actual wire format uses binary prefixes and variable-length integers; the example describes the choice of representation rather than the bytes on the network.

The dynamic table

The static table cannot contain application-specific or changing values, so each direction of an HTTP/2 connection also has a dynamic table. This is a small, connection-local cache of recently transmitted header fields.

When the encoder sends a header as a literal with indexing, it tells the decoder to add that name and value to the dynamic table. Later header blocks can refer to the entry by index. A request containing a long cookie, for example, can pay its full cost once and then use an index on subsequent requests.

The dynamic table is ordered with the newest entry first. Its indexes follow the static-table indexes, so an index identifies one entry in the combined static-and-dynamic view. Entries are evicted from the oldest end when the table exceeds its capacity.

Capacity is measured in bytes using HPACK's accounting rule: the length of the name plus the length of the value plus 32 bytes of overhead per entry. HTTP/2 settings let the decoder advertise the maximum dynamic-table size it will accept. A capacity of zero effectively disables dynamic storage. The encoder can also reduce the active size, causing entries to be evicted.

Three useful representations

For each header field, HPACK can choose among several forms:

There is also a never-indexed literal form. It communicates that a value should not be stored for later reuse, which is useful for sensitive or highly variable fields. The choice is made separately for each header field; HPACK does not assume that every field deserves caching.

An encoder may reuse a name from either table and send only a new value. If the name is not present, it sends the name as text.

Huffman coding for remaining text

Table indexes eliminate repeated fields, but some names and values still have to be transmitted. HPACK can encode each such string using a fixed Huffman code defined by HPACK itself. Huffman coding assigns shorter bit sequences to common characters and longer sequences to uncommon ones, so typical header text takes fewer bits.

Each encoded string indicates whether Huffman coding was used and includes its encoded length. The decoder reads the bits using the shared code table. Padding at the end is defined so the decoder can identify the end safely.

Huffman coding handles character-frequency repetition within a string. It does not replace the dynamic table: the table handles repeated fields across requests, while Huffman coding makes the text that remains smaller.

How the endpoints stay synchronized

The dynamic table works only if both sides maintain the same state. For one direction, the endpoint sending headers is the encoder, and the endpoint receiving them is the decoder. The reverse direction has a separate encoder and decoder; the two directions do not share one table.

Synchronization follows directly from the wire representations:

  1. The encoder emits a literal with indexing.
  2. The decoder reads that representation and inserts the identical name and value.
  3. Both tables now have the same new entry at the same position.
  4. Both apply the same capacity limit and evict the same oldest entries.
  5. A later indexed representation refers to the same entry number at both ends.

The encoder must therefore never reference an entry that the decoder has not yet processed. Header blocks are decoded in the connection's received order, and table changes take effect as those blocks are processed. HTTP/2 may interleave traffic from multiple streams, but a header block is carried as a complete HEADERS frame sequence before another header block can disrupt that sequence. The encoder's view of the table must follow that same processing order.

HPACK also has explicit table-size-update instructions. These occur at the start of a header block and let the encoder announce a permitted capacity change. The decoder applies the update before interpreting later indexes, keeping eviction behavior deterministic.

When an incident log reports an HPACK decoding error, an invalid index, or a dynamic-table-size violation, it usually means this shared state has diverged or a peer sent a representation outside the negotiated limits. HPACK is not an independently reset cache for each request: it is ordered compression state attached to the HTTP/2 connection.

← All articles