The Two Temples: On the Contrast Between Ephemeral and Persistent Logging
There is a quiet, fundamental schism in the architecture of our small services, a philosophical divide that reveals itself not in the grand design but in the humble log file. It is the choice between two temples: one built to be ephemeral, a sand mandala swept away by the next tide; the other carved in stone, meant to endure as a permanent record. This is the contrast between logging to stdout and logging to a file, and the choice is anything but trivial.
The first temple, the ephemeral one, is built on the principle of disposability. Here, a service simply speaks its mind to the standard output stream, whispering its status updates, errors, and warnings into the void. It trusts that a greater system—a container orchestrator, a process supervisor, a log shipper—is listening intently, ready to capture its transient words and ferry them to a central citadel like Loki or Elasticsearch. This approach is clean and modern. The service itself remains blissfully unaware of disks, rotations, or retention policies. It is a pure expression of its function, unburdened by the concerns of persistence. Its logs are like breath on a cold morning: meaningful in the moment, but gone almost as soon as they are formed, existing only because something else was there to witness them.
The second temple is one of persistence and self-reliance. Here, a service meticulously inscribes its own history directly onto the local filesystem. It manages its own log files, perhaps with the steady rhythm of a log rotator, creating `service.log`, then `service.log.1`, `service.log.2.gz`, and so on, a tangible chronology of its existence. This method is older, quieter. It does not assume a network is always available or that a log aggregator is always listening. It creates its own anchor to reality. In the event of a catastrophic network partition, when the ephemeral logger’s words vanish into silent ether, this service’s story remains, etched into the very machine it runs on, waiting patiently to be discovered.
The Choice of Philosophy
This technical decision is, at its heart, a philosophical one. The ephemeral approach embraces a distributed, cloud-native idealism. It believes in the whole, in the connected system, and is willing to sacrifice individual durability for collective intelligence and operational simplicity. The persistent approach is rooted in a more pragmatic, perhaps even stoic, individualism. It believes a service should be a self-contained unit, responsible for its own truths until explicitly relieved of that duty.
Neither is inherently right or wrong. The ephemeral stream is elegant for orchestrated, scalable environments where a local disk is an implementation detail, not a guarantee. The persistent file is a bastion of reliability for the standalone server, the raspberry pi in a closet, the critical system where every word must be accounted for, even when the network falters. One is a trust in the ecosystem; the other is a trust in oneself. In our quest for reliability, we must choose which temple to build in, understanding that each offers a different kind of quiet assurance.
Notes & further reading
A few pages I came back to while writing this: