The Two Watches: Comparing Synchronous and Asynchronous Logging

There’s a moment in every service’s life when it must decide how to speak. Not to users, but to you, its keeper. It must log. And the method it chooses to report its state—be it a whisper or a shout—can define its character, its resilience, and its relationship with the truth. I’ve come to see this choice not as a technical toggle, but as a philosophical divide between two types of timekeeping: the pocket watch and the sundial.

One approach is the pocket watch: synchronous logging. This is the immediate, undeniable record. The service performs an action, and before it moves a single muscle further, it must stop, open its watch, and etch the time and event into its ledger. The entry is made in lockstep with the event itself. The benefit is a perfect, unassailable truth. When you read that log line, you know with certainty that the event happened precisely then, before anything that follows. It’s the log of a meticulous accountant, leaving no room for doubt in the sequence of operations.

But this fidelity comes at a cost. Stopping the world to make a note introduces latency. Under heavy load, this constant pausing to document reality can become the primary bottleneck, the thing that slows the very reality it’s trying to record. The act of observation changes the outcome. A service logging synchronously can, in its quest for perfect truth, become less of a service and more of a archivist struggling to keep up.

The Sundial's Shadow

The other approach is the sundial: asynchronous logging. Here, the service lives its life at full tilt. It performs its duties, and as it goes, it tosses messages—its observations, its errors, its heartbeats—into a queue or a buffer. Another, separate process, running on its own time, collects these messages and diligently writes them to the permanent log. The service itself never waits. It never pauses. It is blissfully, speedily unaware of the scribe working behind it.

The advantage is raw performance. The service operates with minimal overhead, its flow uninterrupted by the administrative duty of documentation. It feels faster, more responsive. But the record it leaves is different. It’s a story told from memory after the fact, not a live transcript. Under extreme duress, if the service fails catastrophically, those last few crucial messages might still be in the buffer, a story lost because the scribe couldn’t write fast enough. The log becomes a best-effort history, not an immutable ledger.

Neither is inherently right. The pocket watch gives you certainty when you absolutely must know the precise order of a failure, even if it means the world moves a little slower. The sundial gives you a service that flies, accepting that its diary might have a few smudged entries or a missing last page. Choosing between them isn’t about finding the "best" logging; it’s about deciding what kind of truth your service needs to tell, and what you’re willing to trade to hear it.

Notes & further reading

A few pages I came back to while writing this: