The Unbroken Chain: A Practical Guide to Immutable, Append-Only Logging
We talk a lot about the importance of logging, but we rarely discuss the integrity of the logs themselves. A log file is a story, a narrative of events within a system. But what if that story can be edited after the fact? A log that can be altered is a log that can lie, rendering it useless for diagnostics or, worse, for understanding a breach. The solution isn't more logging; it's better logging. It's building an unbroken chain of truth through a simple, powerful technique: the append-only filesystem.
The core idea is disarmingly simple: once a byte is written to the log, it must never be changed or deleted. This immutability is the bedrock of a reliable audit trail. While complex external systems exist to enforce this, a surprisingly robust implementation can be achieved using a fundamental feature available on any Linux system: the append-only file attribute.
This attribute, managed by the chattr command, is a flag set on a file within a Linux filesystem (like ext4). When set, it prevents the file from being deleted or truncated, and data can only be added to the end of it. Even the root user is bound by this rule. To set it up, first create your log file. Then, before your service starts writing, run: sudo chattr +a /var/log/my-service/critical.log. The +a flag is for 'append'.
The Quiet Guardian
Once set, your logging process—be it a custom script, a daemon, or a containerized application—can write to this file normally. It can open the file, seek to the end, and add new lines. The magic happens when something, or someone, tries to interfere. An accidental rm command? Permission denied. A well-intentioned but catastrophic log rotation script that tries to move or truncate the file? Operation not permitted. The log remains, an unwavering witness.
This technique shines in its simplicity and its constraints. It forces a healthy discipline. You cannot rotate this file in the traditional sense. Instead, you must build your logging around it. A common pattern is to use the logrotate utility not to touch the immutable file itself, but to copy its contents for archival and then signal the application to start a new, fresh immutable log file. The old one remains sealed, a perfect snapshot of a period in time.
Implementing this is a small act of architectural defiance against chaos. It is a declaration that some truths must be preserved, unedited. It turns your log from a mutable notepad into a permanent ledger. It’s not a flashy solution, but a profoundly boring and reliable one. In the quiet hum of your servers, that append-only log is a silent sentinel, ensuring that the story it tells is the story that actually happened, one immutable line at a time.
Notes & further reading
A few pages I came back to while writing this: