TLB & Page Fault Handling
Virtual Memory Internals · OS Core Internals track
5 min read
In this module
The TLB: Caching Address Translations
A four-level page table walk needs four memory accesses just to translate a single virtual address. That's before the actual data access even begins. If every memory reference paid that bill, programme performance would crater.
The hardware solves this with the Translation Lookaside Buffer (TLB), a small, extremely fast cache inside the CPU that stores recent virtual-to-physical translations. The CPU checks the TLB first on every virtual address. If the translation is there (a TLB hit), no page table walk is needed.
TLB Structure
The TLB is built from content-addressable memory (CAM): hardware that searches every entry in parallel by content rather than by index. Present a virtual page number and the TLB instantly tells you whether it has a matching entry.
Modern x86-64 processors have a multi-level TLB:
- L1 DTLB (data) - approximately 64 entries, split between 4 KB and 2 MB page translations. Accessed in a single CPU cycle.
- L1 ITLB (instruction) - similar size, dedicated to instruction fetches
- L2 TLB (unified) - approximately 1536 entries, shared between data and instruction translations
The walker below adds a 4-entry TLB to the scaled-down four-level machine from the previous lesson (4-bit indices, 4 KB pages). Translate 0x0C2805A0: the TLB misses and the walk runs all four levels. Translate it again and the TLB answers straight away. Now try 0x0C280FFF, a different byte in the same page, then 0x0C2815A0, which is the next page over. Only the page number is cached, so the first hits and the second walks again.
Page Table Walker (4-level, 4096B pages)
The TLB caches virtual-to-physical translations in content-addressable memory, skipping the four-level page table walk for recently accessed pages. A TLB miss triggers a hardware page table walk; a page fault fires when the walk finds no valid mapping, or a mapping that forbids the access.
TLB Misses
When the TLB doesn't have the translation (a TLB miss), the CPU's page table walker hardware traverses the four-level page table in memory. If the walk succeeds (page is present and accessible), the translation is loaded into the TLB and the memory access proceeds.
TLB misses are expensive (roughly 10-30 nanoseconds for the page table walk), but they're cheap compared to a page fault.
TLB Shootdowns
On multi-core systems, each core has its own TLB. When the kernel modifies a page table entry (unmapping a page, changing permissions), stale TLB entries on other cores become dangerous. A core using an outdated translation could read freed memory or sail past new restrictions.
The kernel handles this with a TLB shootdown: it fires an Inter-Processor Interrupt (IPI) at every other core, telling them to invalidate the affected TLB entries. It's a synchronisation bottleneck. All cores have to stop, invalidate, and acknowledge before the kernel can proceed.
TLB shootdowns are a real source of overhead for workloads that hammer the page tables, like memory-intensive applications or some virtualisation scenarios.
Page Faults
When the page table walk finds the Present bit clear in the page table entry, the CPU raises a page fault, an exception that traps into the kernel. It raises one too when the page is present but the access breaks its permissions: a write to a read-only page, user code touching a kernel page, or an instruction fetch from a no-execute page. The kernel's page fault handler decides what to do. Some of these are deliberate. After fork, parent and child share their pages marked read-only, and the first write to one faults so the kernel can copy it (copy-on-write).
Not all page faults are alike:
Minor Faults (Soft Faults)
The page exists in physical memory but isn't currently mapped in this process's page tables. Common case: a page is sitting in the page cache but the process's PTE hasn't been set up yet. Resolution is just updating the page table entry. No disk I/O.
Major Faults (Hard Faults)
The page isn't in physical memory at all. It has to come from disk. The kernel pulls the page from swap or the backing file, allocates a physical frame, reads the data, and updates the page table entry. The cost is brutal:
- HDD: 1-10 milliseconds per fault
- SSD: 50-200 microseconds per fault
During a major fault, the faulting process is put to sleep (TASK_UNINTERRUPTIBLE) while the I/O completes. That's why heavy swapping torches system performance: each major fault is thousands of times slower than a normal memory access.
# Monitor page faults for a specific processperf stat -e page-faults,minor-faults,major-faults ./my_program
# System-wide page fault statisticssar -B 1Why It Matters
For security researchers, TLB behaviour is bedrock. Meltdown, for example, exploited the window in which an Intel CPU would speculatively read a kernel address before the page table's user/supervisor permission check stopped it. The stolen byte was then smuggled out through the data cache with a Flush+Reload timing probe. TLB timing itself is a weapon too: attacks such as EntryBleed time how fast kernel addresses translate in order to defeat KASLR. Understanding the TLB and page fault path also explains why kernel exploit primitives that can rewrite page table entries are so devastating: they hand an attacker control over address translation itself.