Notes on triaging a crash

A crash is a question. Something poked the program in a way it didn't expect; the kernel sent a signal; the process died. You've got the corpse. The interesting bit is what killed it, and whether the killer looks more like a viable attacker than a bored fuzzer.

Most crashes you'll see in the wild come from a fuzzer chewing on input or from a manual experiment that overran something. Either way, the workflow is the same: capture the evidence, classify the crash, decide whether to keep going. The biggest mistake junior researchers make is leaping straight to "is it exploitable?" before they've even confirmed which instruction crashed.

Capturing the crash

Step one is getting a reproducible artefact. A crash you can't reproduce is a war story, not a bug.

If the crash happens out-of-process and you have the input that triggered it, save the input first. Hash it. Store it somewhere named after the hash so future-you can find it. Fuzzers like AFL++ and libFuzzer already do this; if you're triaging by hand, mimic the discipline. The number of "exploitable" findings that turned out to be irreproducible because nobody saved the seed is depressing.

Next, get a core dump. On most Linux distros core dumps are disabled by default. Switch them on for the shell session:

ulimit -c unlimited
echo "/tmp/core.%e.%p" | sudo tee /proc/sys/kernel/core_pattern

Run the crash again. You should get /tmp/core.<binary>.<pid>. Open it in gdb:

gdb /path/to/binary /tmp/core.binary.12345

If the program is interactive or daemonised, attach gdb live with gdb -p <pid> before triggering the crash, set set follow-fork-mode child if needed, and let it stop at the signal. For services that auto-restart, systemd's coredumpctl list and coredumpctl gdb are the path of least resistance.

Reading the corpse

The first thing to look at is info registers and the faulting instruction. Use x/i $rip (or $pc on aarch64) to see what was about to execute. The instruction tells you almost everything you need for the first triage pass.

=> 0x4011a3 <process_record+45>: mov BYTE PTR [rax],dl

That's a write to the address in rax. Check rax. If it's something like 0x4141414141414141 (the classic AAAAAAAA pattern of unchecked attacker bytes interpreted as a pointer), you have an attacker-controlled write target. If it's 0x0 or 0xdeadbeef, you have a write to a dud address; less interesting.

Ask three small questions:

1. Read or write? A mov [reg], val is a write. A mov reg, [val] is a read. A call or jmp to a controlled address is a control-flow primitive. Reads are generally less exploitable than writes, with the obvious exception of info-leak primitives that defeat ASLR.
2. Stack or heap? Look at $rsp and the addresses near the crash. Anything in the form 0x7fff... is usually stack. 0x55... or 0x7f...8000 is more often heap or a library mapping. info proc mappings confirms it.
3. Controlled or uncontrolled? Trace back from the faulting register to where its value came from. If it's a copy of attacker input, you have a primitive. If it's an arithmetic mangle of a length field that overflowed, you might also have a primitive, just one stage removed.

Classifying the crash

The decision tree above is the rough triage I run mentally on every crash. None of the boxes are absolute; an info leak in the wrong place is gold, and a partial write at the wrong offset can still escalate. Use it as a default, not a verdict.

The "exploitable" tool and why it lies to you

There's a venerable GDB plugin called exploitable (originally from CERT, ported and maintained in various forks) that classifies a crash with a label: EXPLOITABLE, PROBABLY_EXPLOITABLE, UNKNOWN, NOT_EXPLOITABLE. People love it because it produces a verdict and that verdict is easy to put in a triage report.

It's also wrong a lot. The plugin uses a ruleset based on the faulting instruction class, the signal type, and a small amount of register inspection. A crash on call [rax] with a controlled rax lights up green. A use-after-free that crashes deep inside a glibc heap function might be marked unknown when it's a perfectly reachable RCE primitive. A stack canary failure might be marked exploitable when in practice it's the canary doing its job and the bug is unreachable past the abort.

Use exploitable as a tie-breaker on a large fuzz corpus, not as a verdict on a specific crash. For real triage, modern fuzzing harnesses (AFL++ has its own triage_crashes.sh, sanitisers like ASan and UBSan are radically more useful than the old plugin) plus targeted scripting on the corpus give you something defensible.

ASan in particular is the single biggest upgrade over exploitable-style heuristics. Compile your target with -fsanitize=address if you can, replay the crashing input, and ASan reports the bug class (heap-buffer-overflow, use-after-free, double-free, stack-buffer-overflow) in plain English, with the allocation and freeing sites for UAFs. That's a triage report in itself.

Is it worth chasing?

Decide using a small number of factors. None of these is sufficient on its own; weigh them together.

• Reachability. Can an unauthenticated attacker reach this code path through a normal interface? If the only way to trigger the crash is to feed /dev/random into a debug interface enabled by --enable-fuzz, you have a fuzzer artefact, not a vulnerability.
• Primitive quality. Controlled-write-anywhere is gold. Controlled-call is gold. Controlled-read of a pointer is good. NULL deref in a privileged process is sometimes good (if you can spray mmap low addresses, on systems that allow it). NULL deref in user-space on a hardened kernel is almost always nothing.
• Mitigation cost. Even the cleanest primitive is worth less in the presence of CFI, shadow stacks, ASLR, and the rest. Cost the exploit roughly. If it would take three months of work to bypass the mitigations, that's a signal about whether to keep going.
• Bug class novelty. A new bug class in a hardened target beats a familiar bug class in a soft one, both for reputation and for the depth of what it teaches.

Most crashes fail at the first hurdle: not reachable, no primitive, nothing to chase. Discard them quickly. The few that survive to the second pass deserve real time.

What to write down

Even for crashes you discard, log the basics: the input hash, the faulting instruction, the bug class (your guess), the mitigations on the target, the date. Future you will thank present you when the same fuzzer dredges up the same artefact for the fifth time and you can mark it duplicate without re-triaging.

The triage notes that survive long-term tend to look more like a lab notebook than a bug report. Short, dated, full of register dumps and one-line hypotheses. They're the connective tissue between "I have crashes" and "I have an exploit", and most researchers underestimate how much of the work happens there.

Where this fits in

Module 83: Binary Forensics & Crash Analysis walks through core-dump archaeology in depth, including the kernel-side machinery that produces them, the gdb commands that aren't on the cheat sheets, and the heap-state reconstruction techniques that turn an opaque libc backtrace into a story you can tell. Worth a session before you start triaging at scale.