NIC Interrupt Coalescing and Packet-Batching Tradeoffs
NIC interrupt coalescing is a network-interface technique that waits for several packet events, or for a short time window, before interrupting the CPU. It reduces the number of CPU interruptions needed to process network traffic, at the cost of delaying notification about some packets.
It is also called interrupt moderation. You may encounter it in a NIC driver setting, an ethtool configuration dump, a performance investigation, or an incident where lowering latency made CPU usage rise.
The problem: one interrupt per packet
A network interface card (NIC) receives Ethernet frames from the wire. It places the packet data into memory using DMA, or direct memory access, so the NIC can transfer bytes without the CPU copying every byte itself. It also records what happened in a shared data structure called a receive ring: a circular list of entries describing packet buffers and their completion status.
The CPU still needs to know when new packets are ready. The NIC normally does this by raising a hardware interrupt, a signal that makes a CPU pause its current work and run a driver handler.
If the NIC raises one interrupt for every packet, the system pays that fixed cost repeatedly:
- the CPU switches into interrupt-handling code;
- the driver examines completed receive entries;
- packet-processing work disturbs the CPU's current instruction and cache state; and
- the system may spend more time handling notifications than processing packet data.
At low traffic rates, this may be acceptable. At high packet rates, especially with many small packets, the interrupt rate can consume substantial CPU capacity even when the total number of bytes is modest.
What the NIC batches
Interrupt coalescing does not usually combine several packets into one larger packet. The packets remain separate in memory and in the receive ring. Instead, the NIC combines their notifications.
A typical receive sequence looks like this:
- The NIC receives a packet and uses DMA to write it into a buffer.
- It marks a receive-ring entry complete.
- Instead of interrupting immediately, it starts or continues a coalescing interval.
- It interrupts when a configured packet count is reached, when a configured timer expires, or when another hardware-specific condition occurs.
- The driver handles the notification and processes several completed entries in one visit.
The driver may process the entries directly in the interrupt handler. On Linux, a common design is NAPI, a mechanism that briefly uses an interrupt to start polling: the driver disables repeated receive interrupts and then processes a bounded batch of ready entries. Here, polling means checking the receive ring in a controlled loop; it does not necessarily mean continuously asking the NIC when no traffic exists.
The important boundary is this: the NIC decides when to notify, while the operating-system driver decides how to process the work after notification. Coalescing makes each notification more valuable by giving the driver several packets to handle at once.
The two useful thresholds
Hardware commonly exposes two receive-side controls:
- A frame or packet threshold: interrupt after roughly this many completed receive events.
- A time threshold: interrupt after roughly this many microseconds have elapsed since the relevant receive event or coalescing interval began.
The exact names and semantics vary. A setting called rx-frames may count receive descriptors or completion events rather than application-level packets, and a setting called rx-usecs is normally expressed in microseconds. Some NICs have separate settings for transmit completions, which notify the driver that outgoing packets have left the device.
In a simplified example, suppose the NIC is configured for a threshold of eight packets or 50 microseconds. If eight packets arrive quickly, the NIC can interrupt after the eighth. If traffic is sparse, it need not wait forever for eight packets: the timer causes an interrupt after about 50 microseconds. The resulting batch might contain only one or two packets.
The timer is an approximate hardware behavior, not a universal promise that every packet waits exactly that long. Whether it starts with the first packet, is restarted by later packets, and how it interacts with other queues depends on the NIC and driver.
The tradeoff: CPU efficiency versus latency
Coalescing changes the number and timing of interrupts, so its central tradeoff is straightforward:
| More coalescing | Less coalescing |
|---|---|
| Fewer interrupts | More interrupts |
| Lower notification overhead | Higher notification overhead |
| Larger batches and better amortization | Smaller batches and faster notification |
| More receive latency and possible jitter | Lower receive latency |
A larger packet threshold or longer timer usually improves throughput efficiency. One driver visit can inspect many completed entries, reducing per-interrupt work and often improving cache locality. This is useful for bulk transfers, storage traffic, and high packet-rate workloads where CPU utilization matters more than the first few microseconds of latency.
The cost is notification latency: time between the NIC receiving a packet and the CPU beginning to handle it. A packet that arrives just after a notification may wait for the next packet threshold or timer. That delay can affect request-response protocols, small RPCs, market-data systems, online games, and other workloads that care about tail latency rather than only total throughput. Batching can also produce burstier delivery to the network stack and application.
A small threshold or short timer reduces that waiting time, but the CPU handles more interrupts and smaller batches. At the extreme, disabling coalescing approaches one notification per packet and can waste CPU. Increasing it too far can make the system efficient at processing large batches while making short interactions noticeably slower.
Choosing and interpreting settings
There is no universally best value. The right setting depends on packet size, packets per second, CPU headroom, queue count, and whether the workload is throughput- or latency-sensitive. Modern NICs and drivers may offer adaptive coalescing, which changes thresholds based on observed traffic: more moderation under heavy load and less under light or latency-sensitive load.
When diagnosing a setting, measure more than bandwidth. Compare CPU utilization, packets per second, interrupt rate, application latency, and especially high-percentile latency. Also check whether receive-ring pressure or packet drops change. A coalescing adjustment can solve an interrupt-CPU problem while creating a latency problem—or do the reverse.
The practical mental model is simple: the NIC is not delaying the packet's arrival on the wire; it is delaying the CPU's notification that one or more packets are ready. The thresholds decide how much notification overhead the system pays in exchange for that delay.