The Two Temples: On the Contrasting Devotions of Logging and Metrics
There is a quiet war fought in the basements of our systems, a conflict of faith between two schools of thought on how to know a machine. One builds a temple of words, the other a cathedral of numbers. I came to understand this not through a grand failure, but through a slow, dawning realization that my own faith had been misplaced, that I was praying to the wrong god for the wrong kind of deliverance.
For years, my devotion was to the Log. I believed in the sanctity of the line, the narrative. A service would speak to me through its carefully crafted messages, telling a story of its day. I would tail its output, a priest listening for confession, seeking the specific sin that caused a stumble. The log was my detailed chronicle, a journal of events with context and nuance. When the ‘order processing’ service began to stutter, I would grep for ‘ERROR’, then ‘WARN’, and piece together the timeline from its literary fragments. It felt like detective work, human and intimate.
But the Metrics people worshipped a different ideal. Theirs was a faith of aggregation, of trends, not tales. They built dashboards not of words, but of graphs—spiky lines representing throughput, latency, and error rates. They didn’t care about the specific complaint of a single failed transaction; they cared that the 99th percentile latency had doubled in the last hour. Their truth was statistical, a view from ten thousand feet where individual events blur into a pattern. It was a cooler, more distant religion.
The revelation came during a slow degradation, a death by a thousand cuts. The logs were clean. No new errors, no alarming warnings. The service was simply… slower. It felt heavy. It was the metrics, that unwavering graph of p95 latency, that showed the truth I could not see in the logs: a steady, insidious climb. The logs told me nothing was wrong. The metrics showed me everything was.
I see it clearly now. Logging is the practice of history, a record of the particular. It answers the question "What happened?" with a story. Metrics are the practice of prophecy, a reading of the tides. They answer the question "What is happening?" with a shape. One is not superior to the other; they are complementary devotions. The log explains the spike on the graph; the graph reveals the importance of the log entry you just read. To run a small service reliably is to maintain both temples—to listen for both the whispered story and the hum of the sphere.
Notes & further reading
A few pages I came back to while writing this: