Heartbleed, twelve years later

Twelve years ago this month the OpenSSL project shipped a one-line patch that the rest of the internet spent the next several weeks scrambling to deploy. The bug was CVE-2014-0160, branded almost immediately as Heartbleed, and the immediate damage was bad enough that the response reshaped how a small but load-bearing piece of free software gets funded.

Worth revisiting now, partly because the technical lesson is unusually clean, and partly because the institutional changes it triggered have aged in interesting ways.

The bug, in one sentence

OpenSSL's implementation of the TLS heartbeat extension trusted a length field that came from the peer, and used it to decide how many bytes to copy back. There was no check that the requested length matched the data that had actually been sent. Send a heartbeat saying "here is one byte of payload, please echo back 65535 bytes," and you got back one byte of your own data followed by 65,534 bytes of whatever lived next to it in the OpenSSL allocation arena.

The fix was a comparison against the real record length. Two lines, three if you count the brace.

What 64KB of process memory contains

The grim part. SSL termination processes hold the things you would least like to leak: the server's private key, recently decrypted form posts, session cookies, fragments of other users' requests, the contents of the heap arena from a few microseconds ago. None of this is supposed to leave the process. Heartbleed mailed it to anyone who asked, in 64KB blocks, repeatedly, with no entry written to any log because the heartbeat protocol exchange happened below the application layer.

That last detail is worth sitting with. Heartbleed left no fingerprint. A server hit a thousand times in a row by an attacker exfiltrating 64MB of memory looked, from the application's perspective, exactly like a server that was perfectly fine. The first defenders to deploy heartbeat-aware IDS rules did so days after the disclosure, by which point the bug had been in the wild for over two years.

What rotation looks like at internet scale

The patch was the easy half. The hard half was acting on the assumption that every key on every server that had run vulnerable OpenSSL was potentially in someone else's hands. Cloudflare's challenge, where they put a vulnerable server online and asked people to extract the private key, was answered within hours. Once that was confirmed, the question was no longer whether keys had leaked but how many of them.

Certificate authorities issued more new certificates in the week after disclosure than in the previous month. The CRL infrastructure, which had spent years quietly not really working, suddenly got asked to do its job and largely failed. Browsers fell back to OCSP stapling, which had been a niche feature, and treated the absence of a stapled response as suspicious in a way they had not before. The question of how to revoke a TLS certificate at scale, which had been theoretical, became urgent overnight.

A lot of organisations also discovered for the first time that their session cookies, password reset tokens, and API credentials were now strictly speaking compromised. Mass invalidations followed. Users got logged out. People who had used the same password across multiple sites got an unwelcome reminder that this was a bad plan.

OpenSSL's funding problem, briefly visible

The political story is the one most people forget. At the time of the disclosure, the OpenSSL Software Foundation had an annual budget of around $2,000 in donations, plus consulting income, and roughly one full-time-equivalent developer maintaining a library that secured most of the world's HTTPS traffic. The bug had been introduced in 2011 by a volunteer contributor and slipped through code review because there was, effectively, nobody being paid enough to do thorough code review.

This was embarrassing enough that the Linux Foundation set up the Core Infrastructure Initiative within a few weeks, with funding from Amazon, Google, Microsoft, IBM and others. OpenSSL got dedicated paid maintainers for the first time. Other underfunded projects sitting on the same critical path got attention as well. Some of that attention has held up. Some of it quietly evaporated once the news cycle moved on.

OpenSSL itself spent the next two years aggressively pruning its codebase. Dead protocols were removed. The unsupported platforms list got shorter. The tests grew teeth. None of this would have been politically possible without Heartbleed providing cover.

The fork and the rewrite

Two responses to "OpenSSL's codebase is a swamp" are worth tracking.

OpenBSD forked the codebase the week after disclosure and called it LibreSSL. The first commits were a textbook in paranoid C: removing VAX support, removing the bespoke malloc wrapper that had defeated valgrind for years, deleting tens of thousands of lines of code in the first month. LibreSSL is still maintained today. It ships in OpenBSD by default and in macOS for many of the system tools, and it has had its own bugs, but the cleanup itself stands as one of the better demonstrations of what an aggressive code audit looks like.

Google's BoringSSL is the other interesting fork. Less public, more internal, designed to support Google's own infrastructure rather than to be a general-purpose drop-in. BoringSSL is what powers Chrome's TLS stack, Android's, and a fair chunk of the cloud.

The longer-tail response is the one that says you should probably not be writing this stuff in C at all. Rustls, written in memory-safe Rust, is now used by Curl, by parts of AWS, and by a range of language runtimes. It's not a complete OpenSSL replacement and probably never will be, but it's the existence proof that a TLS stack does not have to be a heap-of-pointers archaeological dig.

What still hasn't changed

The bug class is reading past the end of a buffer because the length field is attacker-controlled. It's older than TLS, older than C, older than most of the people writing C today. Memory-safety mitigations have made it harder to weaponise, but they have not made it harder to introduce. A new Heartbleed, in some other corner of some other widely-deployed C codebase, is not implausible. The watch is on for it.

What has changed is the response apparatus. CVE coordination is faster. Cert rotation is more or less automated thanks to Let's Encrypt and ACME. The expectation that a critical disclosure will land with a patch already shipped, an advisory written, and a rollout plan attached is now standard. None of that existed in the form it has today before April 2014.

Where this fits in

For the protocol-level picture (what TLS actually does, where the heartbeat extension fit into the handshake, how a length-prefixed record is supposed to be parsed), Module 28 (Cryptographic Protocols) is the place to start. For the implementation-level pitfalls that turn a clean spec into a bleeding server, Module 29 (Applied Crypto and Pitfalls) walks through the lessons that the Heartbleed era forced into the curriculum.