The Humble Receipt: How to Build a Simple, Unbreakable Audit Trail
We spend a lot of time thinking about backing up our data, but far less on backing up our understanding. When a user reports a problem, when a transaction seems to vanish, or when a system hiccups in the night, the question is rarely "where is the data?" but rather "what happened to it?" We need a story. We need an audit trail.
Logs are the obvious answer, but they are often a firehose of technical ephemera, a roaring river of events where the one crucial pebble is impossible to find. The solution isn't more logging; it's better logging. It's creating a specific, deliberate, and immutable record of business-level events. I call this building a receipt.
The technique is simple. For every meaningful action your service performs—a user signup, a file upload, a payment processed, a configuration change—you write a single, self-contained line to a dedicated file. This is not a debug log. It is a formal receipt. Its format is rigid and unchanging: a timestamp, a unique event ID, the actor (user or system), the action, and the key identifiers involved. For example: 2023-10-27T14:32:01Z | TXN-7A3B9C | system:cron | archived_project | project_id:845
The power of the receipt lies in its simplicity and its separation. It lives in its own flat file, appended to by a single, dumb process. It is not queried in real-time; it is the source of truth you turn to only when you must. Because it is decoupled from your main database and application logic, it becomes a standalone witness. If your database is corrupted or a bug scrambles a transaction, the receipt file remains, a pristine ledger of what was supposed to have occurred.
The Unbreakable Part
To make this trail truly unbreakable, pair it with a primitive but powerful tool: append-only permissions. On a Unix-based system, you can set the immutable attribute on the receipt file, making it impossible to delete or modify without first changing the attribute with elevated privileges. This isn't about keeping out attackers; it's about protecting the record from ourselves—from an accidental rm command in a moment of frustration or a misguided script.
This practice is boring. It is profoundly unsexy. There are no complex streaming platforms or machine learning algorithms involved. It is the digital equivalent of a shopkeeper's carbon-copy receipt book. But in its dull, methodical consistency, it provides a quiet certainty. When the questions come, as they always do, you can reach for this one file and know, with certainty, what was done. It is the story you wrote for your future self, and it never forgets a detail.
Notes & further reading
A few pages I came back to while writing this: