Skip to main content
Be Bitwise

TLB & Page Fault Handling

Virtual Memory Internals · OS Core Internals track

5 min read

Part ofSystems Programming Companion, week 10

In this module

The TLB: Caching Address Translations

A four-level page table walk needs four memory accesses just to translate a single virtual address. That's before the actual data access even begins. If every memory reference paid that bill, programme performance would crater.

The hardware solves this with the Translation Lookaside Buffer (TLB), a small, extremely fast cache inside the CPU that stores recent virtual-to-physical translations. The CPU checks the TLB first on every virtual address. If the translation is there (a TLB hit), no page table walk is needed.

TLB Structure

The TLB is built from content-addressable memory (CAM): hardware that searches every entry in parallel by content rather than by index. Present a virtual page number and the TLB instantly tells you whether it has a matching entry.

Modern x86-64 processors have a multi-level TLB:

  • L1 DTLB (data) - approximately 64 entries, split between 4 KB and 2 MB page translations. Accessed in a single CPU cycle.
  • L1 ITLB (instruction) - similar size, dedicated to instruction fetches
  • L2 TLB (unified) - approximately 1536 entries, shared between data and instruction translations

The walker below adds a 4-entry TLB to the scaled-down four-level machine from the previous lesson (4-bit indices, 4 KB pages). Translate 0x0C2805A0: the TLB misses and the walk runs all four levels. Translate it again and the TLB answers straight away. Now try 0x0C280FFF, a different byte in the same page, then 0x0C2815A0, which is the next page over. Only the page number is cached, so the first hits and the second walks again.

Page Table Walker (4-level, 4096B pages)

Bit-field breakdown
VPN1
1100
= 12 (0xC)
VPN2
0010
= 2 (0x2)
VPN3
1000
= 8 (0x8)
VPN4
0000
= 0 (0x0)
Offset
010110100000
= 1440 (0x5A0)
TLB (0/4 entries)
Empty
Level 1 Page Table
0→PPN 0x16
1→invalid
2→invalid
3→PPN 0x62
4→PPN 0xBE
5→PPN 0xF6
6→PPN 0x2A
7→PPN 0x82
8→PPN 0x56
9→PPN 0x2E
10→PPN 0xFA
11→PPN 0x22
12→PPN 0x4E
13→invalid
14→invalid
15→PPN 0xC2
Level 2 Page Table
0→PPN 0x32
1→PPN 0xFA
2→PPN 0x26
3→PPN 0x4E
4→PPN 0x92
5→PPN 0x1A
6→PPN 0xCE
7→invalid
8→PPN 0x72
9→PPN 0x4A
10→PPN 0xE6
11→invalid
12→PPN 0x52
13→PPN 0x6A
14→PPN 0x3E
15→PPN 0xE6
Level 3 Page Table
0→PPN 0x4E
1→PPN 0xE6
2→invalid
3→PPN 0xAA
4→invalid
5→PPN 0x3E
6→PPN 0xE2
7→invalid
8→PPN 0x1E
9→PPN 0x66
10→PPN 0xC2
11→PPN 0xFA
12→PPN 0xB6
13→PPN 0x4E
14→PPN 0x22
15→invalid
Level 4 Page Table
0→PPN 0x6A
1→PPN 0xC2
2→PPN 0xFE
3→PPN 0xB6
4→PPN 0x4A
5→PPN 0x22
6→PPN 0x86
7→PPN 0xEE
8→PPN 0x3A
9→invalid
10→PPN 0x2E
11→PPN 0xD6
12→PPN 0x9A
13→invalid
14→invalid
15→PPN 0xBE
Ready
Key Concept

The TLB caches virtual-to-physical translations in content-addressable memory, skipping the four-level page table walk for recently accessed pages. A TLB miss triggers a hardware page table walk; a page fault fires when the walk finds no valid mapping, or a mapping that forbids the access.

TLB Misses

When the TLB doesn't have the translation (a TLB miss), the CPU's page table walker hardware traverses the four-level page table in memory. If the walk succeeds (page is present and accessible), the translation is loaded into the TLB and the memory access proceeds.

TLB misses are expensive (roughly 10-30 nanoseconds for the page table walk), but they're cheap compared to a page fault.

TLB Shootdowns

On multi-core systems, each core has its own TLB. When the kernel modifies a page table entry (unmapping a page, changing permissions), stale TLB entries on other cores become dangerous. A core using an outdated translation could read freed memory or sail past new restrictions.

The kernel handles this with a TLB shootdown: it fires an Inter-Processor Interrupt (IPI) at every other core, telling them to invalidate the affected TLB entries. It's a synchronisation bottleneck. All cores have to stop, invalidate, and acknowledge before the kernel can proceed.

TLB shootdowns are a real source of overhead for workloads that hammer the page tables, like memory-intensive applications or some virtualisation scenarios.

Page Faults

When the page table walk finds the Present bit clear in the page table entry, the CPU raises a page fault, an exception that traps into the kernel. It raises one too when the page is present but the access breaks its permissions: a write to a read-only page, user code touching a kernel page, or an instruction fetch from a no-execute page. The kernel's page fault handler decides what to do. Some of these are deliberate. After fork, parent and child share their pages marked read-only, and the first write to one faults so the kernel can copy it (copy-on-write).

Not all page faults are alike:

Minor Faults (Soft Faults)

The page exists in physical memory but isn't currently mapped in this process's page tables. Common case: a page is sitting in the page cache but the process's PTE hasn't been set up yet. Resolution is just updating the page table entry. No disk I/O.

Major Faults (Hard Faults)

The page isn't in physical memory at all. It has to come from disk. The kernel pulls the page from swap or the backing file, allocates a physical frame, reads the data, and updates the page table entry. The cost is brutal:

  • HDD: 1-10 milliseconds per fault
  • SSD: 50-200 microseconds per fault

During a major fault, the faulting process is put to sleep (TASK_UNINTERRUPTIBLE) while the I/O completes. That's why heavy swapping torches system performance: each major fault is thousands of times slower than a normal memory access.

bash
# Monitor page faults for a specific process
perf stat -e page-faults,minor-faults,major-faults ./my_program
# System-wide page fault statistics
sar -B 1

Why It Matters

For security researchers, TLB behaviour is bedrock. Meltdown, for example, exploited the window in which an Intel CPU would speculatively read a kernel address before the page table's user/supervisor permission check stopped it. The stolen byte was then smuggled out through the data cache with a Flush+Reload timing probe. TLB timing itself is a weapon too: attacks such as EntryBleed time how fast kernel addresses translate in order to defeat KASLR. Understanding the TLB and page fault path also explains why kernel exploit primitives that can rewrite page table entries are so devastating: they hand an attacker control over address translation itself.