x86 Instruction Set
x86 Architecture Deep Dive · CPU Architectures track
6 min read
In this module
Variable-Length Complexity
Our teaching ISA uses fixed 16-bit instructions: every instruction is the same size, fetch and decode are dead simple. x86 went the other way. Instructions range from 1 byte to 15. That single encoding decision is one of the most consequential in computing history.
Variable-length encoding lets simple instructions stay compact (a single-byte NOP or RET) while complex instructions with big immediates or multiple operands sprawl across more bytes. Code is usually denser than with a plain 32-bit fixed-width scheme, though compressed RISC encodings such as Arm's Thumb-2 and RISC-V's C extension come close. The price is that the CPU has to do real work just to figure out where each instruction begins and ends.
Instruction Encoding Structure
An x86 instruction can include several optional components:
- Legacy prefixes (0-4 bytes) - modify instruction behaviour (operand size override, repeat, segment override).
- REX prefix (0-1 byte) - extends registers and operand sizes for 64-bit mode.
- Opcode (1-3 bytes) - specifies the operation.
- ModR/M byte (0-1 byte) - encodes register and addressing mode information.
- SIB byte (0-1 byte) - Scale-Index-Base for complex addressing.
- Displacement (0-4 bytes) - memory offset.
- Immediate (0-4 bytes, or 8 for the one form of
movthat loads a full 64-bit constant into a register) - constant value embedded in the instruction.
The decoder has to parse all of this before it even knows what operation to perform. Modern x86 processors burn enormous transistor budgets on instruction decoding alone.
Worked Example: Decoding Five Bytes by Hand
Here are five bytes from a real program: 48 8b 44 cb 08. Let's take them apart the way the decoder does.
48 is a REX prefix. Any byte from 40 to 4f is a REX prefix in 64-bit mode. Its low four bits are flags called W, R, X and B. 48 has only W set, which means "64-bit operand size".
8b is the opcode: mov with a register destination and a register-or-memory source. On its own it doesn't say which register or which memory. That's the next byte's job.
44 is the ModR/M byte. Split it into 2 + 3 + 3 bits: 01 000 100.
mod = 01: the source is memory, with a 1-byte displacement on the end.reg = 000: the destination register is number 0, which israx.rm = 100: this particular value means "a SIB byte follows".
cb is the SIB byte. Split it the same way: 11 001 011.
scale = 11: multiply the index by 8.index = 001: register 1,rcx.base = 011: register 3,rbx.
08 is the 1-byte displacement: add 8.
Put it together and you get:
mov rax, QWORD PTR [rbx + rcx*8 + 0x8]That's "load element rcx from an array of 8-byte values, starting 8 bytes into the structure at rbx". A field access and an array index in one instruction, encoded in five bytes. Register numbers follow the order rax, rcx, rdx, rbx, rsp, rbp, rsi, rdi for 0 to 7, which is worth memorising if you ever plan to read hex dumps.
Here are the same five bytes laid out as the fields the decoder carved them into:
Each field's presence depends on the one before it: the opcode says a ModR/M byte follows, rm = 100 says a SIB byte follows, and mod = 01 says a 1-byte displacement ends the instruction. That chain of "what comes next" is why x86 decoding is hard to do quickly.
Predict: decode 48 89 e5. Opcode 89 is mov with the register-or-memory operand as the destination.
mov rbp, rsp. 48 makes it 64-bit. ModR/M e5 is 11 100 101: mod = 11 means both operands are registers, reg = 100 is register 4 (rsp, the source here) and rm = 101 is register 5 (rbp, the destination). You'll find these three bytes at the start of almost every unoptimised x86-64 function, right after push rbp (55).
Common Instructions
These are the instructions you will encounter most when reading x86 assembly:
| Instruction | Purpose |
|---|---|
MOV dst, src | Copy data from source to destination |
PUSH val | Push a value onto the stack (decrement RSP, write value) |
POP dst | Pop the top of the stack into destination (read value, increment RSP) |
ADD dst, src | Add source to destination |
SUB dst, src | Subtract source from destination |
CMP a, b | Subtract b from a, set flags, discard result |
JMP addr | Unconditional jump |
JE / JNE / JG / JL | Conditional jumps based on flags |
CALL addr | Push return address, jump to function |
RET | Pop return address, jump back to caller |
LEA dst, [addr] | Compute address, store it (not the value at it) |
NOP | Do nothing (one byte: 0x90) |
INT n | Trigger software interrupt number n |
Addressing Modes
x86 supports a rich set of ways to specify operands:
- Immediate -
MOV RAX, 42- the value 42 is encoded directly in the instruction. - Register -
MOV RAX, RBX- the value comes from a register. - Direct -
MOV RAX, [0x1000]- load from a fixed memory address. - Register indirect -
MOV RAX, [RBX]- load from the address held in RBX. - Base + offset -
MOV RAX, [RBP - 8]- common for accessing local variables on the stack. - SIB (Scale-Index-Base) -
MOV RAX, [RBX + RCX*4 + 8]- used for array indexing where the scale (1, 2, 4, or 8) matches the element size. - RIP-relative -
MOV RAX, [RIP + 0x2f1a]- an offset from the current instruction. Almost all global variables in 64-bit position-independent code are reached this way, which is what lets the program load at any address.
The Complexity Tax
All this flexibility costs silicon. Intel and AMD spend a huge slice of their transistor budget on the front-end: the circuitry that fetches, decodes, and shatters x86 instructions into simpler internal micro-operations. Pop the lid on a modern x86 CPU and the execution engine looks oddly RISC-like underneath. The CISC instruction set is, at this point, mostly a compatibility wrapper kept alive for the sake of the existing software ecosystem.
Variable length has a security side too. Because nothing marks where an instruction starts, the same bytes can decode differently depending on where you begin reading. Jump into the middle of a long instruction and the CPU happily decodes its tail as something new. Take a perfectly ordinary mov:
offset 0: b8 5f c3 00 00 mov eax, 0xc35f
offset 1: 5f pop rdi c3 retStart one byte in and the immediate value turns into pop rdi; ret, a classic gadget for loading the first argument register. Attackers lean on exactly this to find useful instruction fragments that the compiler never meant to emit, which you'll meet properly in Return-Oriented Programming.