The Unreliable Witness: On the First Time I Caught a Log in a Lie
We build our small services on a foundation of faith. We trust that when a process says it has written a file, the file is there. We believe that a status code of 200 means a user saw the page we intended. And above all, we have an unshakable conviction in the sanctity of the log file. It is our impartial observer, our black box recorder, the ultimate arbiter of truth in a chaotic system. We are taught to log everything, to trust the logs implicitly, and to follow their narrative without question. This is a dangerous faith.
I learned this the hard way, chasing a ghost in a payment service. The logs were clear, sequential, and utterly wrong. They stated, in unambiguous timestamps and crisp INFO levels, that a critical transaction had been processed successfully. The database, however, told a different story—a story of a missing row, a phantom operation that never committed. The log was a perfect alibi for a crime that never happened. It wasn’t malicious; it was simply recorded at the wrong point in the code’s flow, a triumphant declaration made a fraction of a second before the entire endeavor collapsed. The log didn’t document reality; it documented an intention.
This experience shattered the myth of the log as an objective source of truth. A log is not a video recording of events; it is a story told by the application itself. It is a narrative, subject to the biases, bugs, and timing of its author. We forget that logging calls are just more code, executing in a specific context. They can be skipped by a rogue ‘return’ statement. They can be misdated by a lagging system clock. They can report values from memory that were never persisted to disk. They lie by omission, by misplacement, and by innocent error.
The common advice is to log more, to add more context, to drown the problem in data. But more logging only gives a more detailed, potentially more convincing, story. It does not guarantee its accuracy. The counterintuitive practice, then, is not to trust, but to verify. To treat the log not as gospel, but as a lead—a single testimony in a larger investigation. The truth emerges from the correlation of multiple, independent witnesses: the application log, the database transaction log, the system metrics, and the network flow logs. When they all agree, you can be confident. When they disagree, you have found the edge of your system’s understanding.
Our role is not to be devout archivists of a single record, but skeptical detectives. We must cultivate a system where the logs can be cross-examined. This means designing services that produce events which can be checked against other parts of the stack, and building a culture that questions the narrative, especially when it seems too clean. The most valuable log entry I ever made was one that stated, "About to perform operation X." Its value wasn't in its success, but in its existence as a checkpoint against which the silence of a subsequent success log could be measured as a failure. The log told the truth that time not by what it said, but by what it didn’t.
Notes & further reading
A few pages I came back to while writing this: