Linux TLB Shootdowns on Multicore Systems
A Linux TLB shootdown is the process of making multiple CPUs discard cached virtual-to-physical address translations after the kernel changes a process’s memory mapping. Linux uses inter-processor interrupts, or IPIs—interrupts sent from one CPU to another—to tell those CPUs to invalidate their stale entries.
The operation exists because each CPU has its own translation cache. Updating the shared page tables is not enough.
The translation a CPU caches
Modern processes use virtual memory: a program accesses virtual addresses, while the machine’s RAM is addressed with physical addresses. A page table is the data structure that records mappings between the two, along with permissions such as readable, writable, or executable.
For example, a page table might say:
virtual address 0x400000 -> physical address 0x125000
permission: readable and executable
Reading that table for every memory access would be far too slow. The processor therefore keeps recently used translations in a translation lookaside buffer (TLB), a small, fast cache of virtual-to-physical mappings and their permissions.
Each CPU core, or more precisely each logical CPU, normally has its own TLB. The entries are not automatically shared just because the CPUs use the same page tables.
The problem with changing a mapping
Suppose a process is running on CPU 0 and CPU 1. Both CPUs have cached this translation:
virtual page V -> physical page P
The kernel then handles munmap, which removes a region from a process, or mprotect, which changes its permissions. It updates the process’s page table so that V is unmapped or points elsewhere.
CPU 0 may notice the change and invalidate its own TLB entry. CPU 1, however, could still have the old entry. If it accesses V, the processor can use that cached entry without consulting the newly changed page table.
That creates a correctness and security problem. After an unmap, CPU 1 might access a physical page that the process no longer owns. After a permission change, it might continue writing to a page that should now be read-only. If the old physical page has been returned to the allocator and reused, the access could even corrupt unrelated data.
A page-table update changes the source of truth; a TLB shootdown removes copies of the old answer.
How a shootdown works
The broad sequence is:
- A CPU changes one or more page-table entries.
- It determines which other CPUs may have cached translations for the affected address space and range.
- It sends those CPUs an IPI requesting a TLB invalidation.
- Each receiving CPU runs a small kernel handler, invalidates the relevant local entries, and reports completion.
- The initiating CPU waits until the required CPUs have finished before treating the change as complete.
An address space here means the set of virtual mappings belonging to a process or, in kernel terms, to a memory-management context. Linux tracks which CPUs are currently using such a context, so it generally does not interrupt every CPU in the machine for every change. A CPU that has never run the affected address space cannot have its translations.
The exact invalidation instruction depends on the processor architecture. On x86, for example, the kernel can invalidate an individual page with INVLPG, or use a broader operation when many mappings changed. Other architectures provide different instructions and rules. Linux hides these details behind architecture-specific TLB-flush code.
The initiating CPU must also invalidate its own stale entries. It may do that directly rather than sending itself an IPI. The page-table writes and invalidations must be ordered correctly: CPUs must not finish the operation while still able to observe an old translation.
Why an IPI makes this expensive
An IPI is not an ordinary function call. The sending CPU asks another processor to interrupt whatever it is doing. The receiver must stop its current execution, enter the kernel, run the TLB invalidation, perform the required ordering and synchronization, and resume its previous work.
The sender often waits for acknowledgements. That turns a local mapping change into a multicore coordination point. Its cost can include:
- Interrupt delivery: getting the request to each target CPU takes hardware and kernel work.
- Execution disruption: every target loses time handling the interrupt and may disturb its instruction and data caches.
- TLB invalidation cost: discarding entries can cause later memory accesses to perform slower page-table walks until the TLB warms again.
- Waiting: the initiating CPU may be unable to proceed until all required CPUs report completion.
- Scaling effects: changing a mapping used by many CPUs can require many IPIs and acknowledgements.
The cost is especially visible in workloads that frequently create and destroy mappings, change page permissions, or update large address ranges. Memory allocators, virtual machines, JIT runtimes, high-rate process creation, and heavily threaded applications can all encounter it.
Why every change does not flush everything
A full flush of every TLB on every mapping change would be needlessly expensive. Linux therefore tries to limit both the CPUs contacted and the virtual-address range invalidated. It can also batch several page-table changes into one flush, so a group of updates pays for one coordination event instead of many.
Modern processors may tag TLB entries with an address-space identifier, such as x86’s PCID. This helps retain translations when switching between processes, reducing some context-switch costs. It does not make shootdowns unnecessary: if a mapping or permission changes, a CPU holding the affected tagged entry still needs to discard or update it.
When a performance profile, kernel trace, or incident report mentions TLB flushes, remote TLB flushes, or TLB shootdowns, it is usually describing this synchronization—not a mysterious failure in the memory allocator. The underlying issue is that multiple CPUs have private cached answers, while the kernel has changed the shared mapping those answers came from.