What mmap does under the hood

mmap looks like a function that maps a file into memory, and that's the example most tutorials lead with. Read the manpage long enough and you realise it's also how malloc gets large allocations, how the dynamic linker pulls in every .so on your system, and how a process can share memory with another without going through pipes. The syscall itself is small. The behaviour you get out of it depends almost entirely on which flags you pass.

The syscall

void *mmap(void *addr, size_t length, int prot, int flags, int fd, off_t offset);

You're asking the kernel to set aside length bytes of virtual address space, with permissions prot (some combination of PROT_READ, PROT_WRITE, PROT_EXEC, or PROT_NONE), and to populate it according to flags. addr is a hint, almost always NULL, and the kernel returns the address it actually chose.

Here's the catch: what the kernel hands back is a region of virtual address space, not necessarily any physical memory. The pages have an entry in the process's page table. The page table entries are likely marked not present. Touch one of those pages and you take a page fault, the kernel allocates a frame, fills it appropriately, and resumes your instruction. That's demand paging, and it's where most of mmap's personality lives.

Two axes: backing and sharing

The flags split along two independent dimensions.

Backing: where do the contents of the pages come from? File-backed mappings (you supplied a real fd) take their content from disk. Anonymous mappings (MAP_ANONYMOUS, fd = -1) are filled with zeros on first touch.

Sharing: who else sees writes? MAP_SHARED means writes are visible to other processes that mapped the same file region, and writes propagate back to the file. MAP_PRIVATE gives you a copy-on-write view: you see the file's contents on read, but the first write to a page replaces the page-table entry with one pointing at a freshly allocated, process-private frame.

The four combinations cover wildly different use cases. MAP_PRIVATE plus a real file is how ld.so loads .text segments — read-only code that the kernel can share between every process running the same library, without anyone able to dirty the pristine version. MAP_SHARED plus a real file is the IPC pattern: two processes see each other's writes through the pagecache. MAP_PRIVATE | MAP_ANONYMOUS is what malloc uses for large allocations. MAP_SHARED | MAP_ANONYMOUS after a fork is shared memory between parent and child without a backing file at all.

Demand paging in slow motion

The first mmap call is cheap: the kernel does some bookkeeping in the VMA structure (a struct vm_area_struct, one per region per process) and returns. The expensive part happens later, scattered across faults, and only for the pages you actually touch. If you mmap a 4 GB file and read the last byte, you've paid for one 4 KB page. Sequential read() would have copied all 4 GB.

This is the pitch for memory-mapped I/O. It also explains why benchmarks comparing mmap to read are tricky: the work simply happens at a different time.

/proc/self/maps before and after

Every Linux process has a file at /proc/self/maps listing every mapped region. Cat it from a shell first to see the static picture:

$ cat /proc/self/maps | head
55a4...  r--p 00000000 fd:01 ... /usr/bin/cat
55a4...  r-xp 00003000 fd:01 ... /usr/bin/cat
55a4...  r--p 0000f000 fd:01 ... /usr/bin/cat
55a4...  r--p 00012000 fd:01 ... /usr/bin/cat
55a4...  rw-p 00013000 fd:01 ... /usr/bin/cat
7f...    r--p 00000000 fd:01 ... /usr/lib/x86_64-linux-gnu/libc.so.6
7f...    r-xp 00029000 fd:01 ... /usr/lib/x86_64-linux-gnu/libc.so.6
...

Five regions for /usr/bin/cat itself (its PT_LOAD segments split by permission), then libc.so.6, then ld-linux.so, then a stack and a heap. Every one of those is the result of an mmap.

Now make a tiny program that maps something explicitly:

#include <fcntl.h>
#include <stdio.h>
#include <sys/mman.h>
#include <unistd.h>

int main(void) {
    int fd = open("/etc/passwd", O_RDONLY);
    void *p = mmap(NULL, 4096, PROT_READ, MAP_PRIVATE, fd, 0);
    printf("mapped at %p, first byte %c\n", p, *(char*)p);
    printf("PID %d, hit enter to exit\n", getpid());
    getchar();
    return 0;
}

Run it, switch to another terminal, and cat /proc/<pid>/maps. You'll see a new line near the top with /etc/passwd and the address printed by your program. Touch nothing else, and only the first 4 KB ever leaves the page cache for your address space.

How malloc uses it

glibc's allocator has two main strategies. Small allocations go onto a heap that grows via brk() (a relic of pre-mmap Unix, still useful because adjusting the program break is one syscall). Allocations above M_MMAP_THRESHOLD (128 KB by default) get their own anonymous private mapping, which free returns to the kernel by munmap. The reason is fragmentation: a 1 MB allocation in the middle of the heap is awkward to give back when freed, whereas a 1 MB private mmap is a discrete region that munmap can release outright.

You can watch this happen with strace -e mmap,munmap,brk ./your-program. Allocate something tiny: only brk. Allocate something bigger than 128 KB: a fresh mmap shows up.

Shared libraries

When ld.so loads libc.so.6, it mmaps the file with MAP_PRIVATE, once per PT_LOAD segment — one read-only mapping for .text, another read-only for .rodata, a writable copy-on-write mapping for .data, and so on. Because every process maps the same physical pages until somebody writes (and nobody writes to .text ever), the kernel only ever needs one copy of glibc's executable code in RAM, no matter how many processes are using it. That sharing is the entire reason shared libraries pay for themselves.

Where this fits

Module 102: Virtual Memory Internals starts from page tables and walks up through VMAs, page faults, the page cache, and how mmap, brk, and madvise map onto the underlying machinery. Once you've drawn the diagram a few times, mmap's flag combinations stop feeling like magic and become a small DSL for asking the kernel exactly what you want.