The Two Watches: On the Difference Between Counting and Telling Time
There are two ways to know if a service is alive. One is to ask it. The other is to watch it live. The distinction is subtle, but in the quiet, humming dark of a server rack, it is the difference between a heartbeat and a breath.
The first approach is the formal check-in. A cron job, a monitoring agent, a scheduled task—it reaches out with a polite, structured query. It pings an endpoint, expects a specific HTTP code, runs a script that must return zero. It is a sentinel asking for the password. This method is clean, deliberate, and easily understood. It creates a tidy event in a log: ‘Health check passed at 03:14:00 UTC.’ It is the watch you consult by pressing a button to illuminate the dial. You are actively seeking the time.
The second approach is passive observation. It does not ask; it listens. It is the stream of entries in an application log, the gentle rhythm of a process writing to its own journal. It is the whisper of network traffic on a expected port, the regular sigh of a database committing a transaction. The service is not answering a question; it is simply going about its business, and its existence is confirmed as a byproduct of that work. This is the watch you hear ticking softly from your wrist. You know it is working without ever having to look.
The Comfort of the Question
The active check is a comfort. It is a contract. We have agreed that if the service is well, it will answer ‘yes’ to this specific call. Its failure is equally clear and dramatic. Silence is a definitive alarm. This is necessary for many things—a load balancer needs this binary signal to route traffic away from the sick. But it is also a performance, a moment staged for an audience of one. A service can be cunning. It can muster the strength to whisper ‘yes’ to the health check even as its world is crumbling internally, a patient insisting they are fine to avoid a trip to the doctor.
The Truth of the Background Hum
The passive observation is often a truer portrait of health. It is harder to fake the rhythm of genuine work. If the log flow from a service suddenly flatlines, but the health checks are still passing, you have discovered a profound and telling dissonance. The actor is still on stage, bowing to the empty seats, long after the play has ended. This method requires a different kind of attention. It asks you not to listen for a specific answer, but to understand the music of the system itself—its cadence, its volume, its timbre. You are not checking the time; you are feeling the sun's position in the sky.
Neither approach is superior. They are complementary lenses. The active check is your scheduled meeting, your formal report. The passive log is the casual conversation in the hallway, the ambient noise of a functioning office. The most resilient systems are watched by both: the sentinel who calls out and receives the expected reply, and the gardener who, simply by tending the plot, knows from the color of the leaves and the feel of the soil that all is as it should be. One counts the seconds. The other tells the time.
Notes & further reading
A few pages I came back to while writing this: