x86 Calling Conventions & ABI
x86 Architecture Deep Dive · CPU Architectures track
7 min read
In this module
The Rules of Function Calls
When one function calls another, both sides need to agree on the protocol. Where do the arguments live? Where does the return value go? Who saves which registers? Those rules are the calling convention, baked into the platform's ABI.
An ABI defines how compiled code interacts at the binary level: calling conventions, register usage, stack layout, data alignment, and system call mechanisms. The ISA tells you what the CPU can do; the ABI tells you what software has agreed to do with it. Mix code built against different ABIs and you'll get crashes or silent wrong answers.
cdecl: The Classic C Convention (32-bit)
The cdecl convention is the default for 32-bit x86 C code:
- Arguments are pushed onto the stack right to left (so the first argument ends up on top).
- The return value goes in EAX.
- The caller cleans up the stack after the call returns.
- Registers EAX, ECX, and EDX are caller-saved (the called function may overwrite them).
- Registers EBX, ESI, EDI, EBP are callee-saved (the called function must preserve them).
It's simple, and it's slow. Every argument means a memory write by the caller followed by a memory read in the callee.
System V AMD64 ABI (64-bit Linux/macOS)
The dominant 64-bit calling convention on Unix-like systems passes arguments in registers for speed:
- First six integer/pointer arguments: RDI, RSI, RDX, RCX, R8, R9.
- Additional arguments spill to the stack.
- First eight floating-point arguments: XMM0-XMM7.
- Return value: RAX (integer) or XMM0 (floating-point).
- Callee-saved registers: RBX, RBP, R12-R15. Everything else may be trashed by the function you call.
- The red zone: 128 bytes below RSP that leaf functions can scribble into without bothering to adjust the stack pointer. Small functions skip stack-frame setup entirely.
- Stack alignment: RSP must be a multiple of 16 at the
callinstruction, so on entry to a function it's 8 bytes off, because of the return address. Hand-written assembly that forgets this tends to crash inside libc, on an SSE instruction that expects aligned memory.
Windows x64 Convention
Windows uses a different 64-bit convention:
- First four integer/pointer arguments: RCX, RDX, R8, R9.
- Additional arguments go on the stack.
- Shadow space: the caller must always allocate 32 bytes of stack space for the first four arguments, even though they are passed in registers. The callee may use this space to spill register arguments.
- Floating-point arguments go in XMM0-XMM3, sharing positions with the integer registers: a
doublepassed second travels in XMM1, and RDX goes unused. - Callee-saved registers: RBX, RBP, RDI, RSI, R12-R15 and XMM6-XMM15. RDI and RSI are the surprise if you're coming from Linux.
- The same 16-byte alignment rule at
call. - No red zone.
| Feature | System V AMD64 | Windows x64 |
|---|---|---|
| First arg | RDI | RCX |
| Register args | 6 | 4 |
| Red zone | 128 bytes | None |
| Shadow space | None | 32 bytes |
| Callee-saved | RBX, RBP, R12-R15 | RBX, RBP, RDI, RSI, R12-R15, XMM6-XMM15 |
Linux system calls use a third arrangement. The call number goes in RAX and the arguments in RDI, RSI, RDX, R10, R8, R9. R10 stands in for RCX because the syscall instruction itself overwrites RCX (with the return address) and R11 (with the flags).
Worked Example: Eight Arguments, Two ABIs
Take a call with more arguments than either convention has registers for:
long f(long a, long b, long c, long d, long e, long g, long h, long i);f(1, 2, 3, 4, 5, 6, 7, 8);On Linux (System V). The first six go in registers: rdi=1, rsi=2, rdx=3, rcx=4, r8=5, r9=6. The last two are pushed right to left, 8 first, then 7. Then call pushes the return address on top. At the first instruction of f, the stack reads:
| Address | Contents |
|---|---|
[rsp] | return address |
[rsp+8] | 7 |
[rsp+16] | 8 |
Step through the System V version below, from the caller's pushes to the end of f's prologue. The addresses are illustrative; the offsets between them aren't.
f(1, 2, 3, 4, 5, 6, 7, 8) on System V
rdi, rsi, rdx, rcx, r8 and r9 already hold 1 to 6.
Higher addresses at the top. Each row is one 8-byte slot. Faded rows sit below rsp: popped, but not erased.
When f returns, the caller runs add rsp, 16 to throw the two pushed arguments away. In System V that cleanup is the caller's job.
On Windows x64. Only four registers: rcx=1, rdx=2, r8=3, r9=4. The caller also reserves the 32-byte shadow space, so at the start of f the arguments in memory begin further up:
| Address | Contents |
|---|---|
[rsp] | return address |
[rsp+8] to [rsp+32] | shadow space for the four register arguments |
[rsp+40] | 5 |
[rsp+48] | 6 |
[rsp+56] | 7 |
[rsp+64] | 8 |
Same source line, same CPU, completely different machine code. A debugger, a decompiler and an exploit all have to know which ABI they're looking at before any of these offsets mean anything.
Predict: on Linux, gcc emits mov eax, 1 just before call printf when you print a double. Why?
It's telling printf how many vector registers hold arguments. For variadic functions like printf, System V says al must hold an upper bound on the number of XMM registers used for arguments. The double travels in xmm0, so al is 1. When no floating-point arguments are passed you'll see xor eax, eax instead, which is also why that line turns up before so many calls to printf.
Now go the other way. Each snippet below makes a call or a system call with small constant arguments. Name the convention and the argument values before you open the answers.
; Snippet Amov edx, 3mov esi, 2mov edi, 1call foo; Snippet Bsub rsp, 0x28mov r8d, 3mov edx, 2mov ecx, 1call fooadd rsp, 0x28; Snippet Cmov eax, 1mov edi, 1lea rsi, [rip+0x2f1a]mov edx, 5syscallAnswers
A is System V, foo(1, 2, 3). RDI, RSI and RDX are the first three argument registers. Writing to edi clears the top half of rdi, which is why compilers load small constants through the 32-bit names.
B is Windows x64, also foo(1, 2, 3), in RCX, RDX and R8. The giveaway is sub rsp, 0x28: 32 bytes of shadow space plus 8 more to get RSP back to a multiple of 16, since the caller's own return address left it 8 bytes off.
C is a Linux system call: write(1, buf, 5). syscall rather than call tells you it goes to the kernel, and RAX holds the call number (1 is write on x86-64 Linux). The arguments are file descriptor 1 (stdout), a buffer addressed relative to RIP, and a length of 5 bytes.
You can try both Linux conventions in one program. _start calls a small function using the System V argument registers, then prints the result with the write system call, whose first three arguments use the same registers.
Why This Matters for Security
Calling conventions are exploit-dev bread and butter. Once you control a buffer overflow you need to know:
- Where the return address sits on the stack (right after the saved base pointer in a standard frame).
- Which registers hold function arguments (so you can stage them for the functions a ROP chain will call).
- How the stack is laid out, so your offsets land where you want them to.
That's why an exploit polished on Linux often needs serious surgery to fire on Windows, even when both targets run the same silicon. Same ISA, different ABI — and the ABI rewrites everything about how the stack is organised and how data flows between functions.