The Quiet Vigil of Richard Feynman and the O-Ring

Our world runs on small services and silent sentinels. We obsess over the five nines, the automated rollbacks, the pager that must never go off. We speak of observability and resilience as if we invented the concepts. But sometimes, the most profound lesson in reliability comes not from a manual or a conference talk, but from a historical figure holding a piece of rubber in his hand, surrounded by politicians and press.

Richard Feynman’s demonstration during the 1986 Rogers Commission investigation into the Challenger space shuttle disaster is a masterclass in what we now call "ops." The official focus was on sophisticated telemetry, complex flight dynamics, and high-level decision-making. But Feynman, the brilliant physicist, cut through the noise. He suspected a simple, boring component: the synthetic rubber O-rings, seals designed to contain burning fuel in the solid rocket boosters. In the freezing temperatures on the morning of the launch, they had failed to flex.

In a now-iconic moment, Feynman placed a sample of the O-ring material into a glass of ice water during a televised hearing. He then compressed it with a simple clamp. When he released it, the material did not spring back. It stayed deformed. The "cold soak" test, conducted not in a lab but in a conference room, was a brutal, elegant piece of logging. It produced a single, undeniable, physical data point: the material's resilience had a lower threshold than its operating requirements. The metrics were there in the engineering data, but Feynman created the alert that everyone could finally hear and see.

We, who tend to systems, can see ourselves in this. How often is the critical signal—the gradual increase in latency, the slow fill of a disk, the slight uptick in errors from a particular service—buried in dashboards, normalized away by averages, or explained by a "known condition"? We build complex architectures but sometimes forget to test the O-rings under our specific, real-world conditions. Feynman didn't trust the summarized report; he demanded the raw experiment of the component itself.

The Vigil is in the Conditions

Feynman’s vigil wasn't just about the component test. It was about understanding the conditions. The engineers at Morton Thiokol had known about the O-ring's sensitivity to cold and had argued against the launch. Their concerns were a form of an alarmed ticket, lost in the management chain's desire to meet a schedule. The system had logging, but it lacked a reliable pipeline to escalate the alert to the ultimate decision-makers.

This is the core of our thankless work: to be the one who asks, "What changes when it's cold?" To instrument not just the service, but the environment it runs in. To ensure that the quiet, technical dissent of a monitoring check or a log analysis has a clear, unassailable path to the console, to the report, to the person who can say "stop." We are not just curators of backups and logs; we are stewards of context. We must be the ones who, when presented with a chart showing "all nominal," remember to ask for the glass of ice water.

The O-ring didn't fail because it was badly made. It failed because its operating environment had shifted outside a parameter that was known, but not heeded. Every day, in our stacks, a hundred small O-rings—a config flag, a library version, a network hop—sit waiting. Our job is to know the temperature of the water they're in, and to have the courage, and the clarity, to demonstrate what happens when it drops.

Notes & further reading

A few pages I came back to while writing this: