The Scribe's Pencil: How to Test Your Alerting System Without Waking Up
There is a special kind of dread familiar to anyone who runs a service. It’s not the panic of a 3 a.m. page, though that is a close relative. It’s the quieter, more insidious fear that when a real catastrophe strikes, the alert might never come. The pager stays silent, the email never arrives, and the first sign of trouble is a public outcry. Your alerting system, your supposed sentinel, has failed at its one job. The worst part? You can’t know if it’s working until it’s too late. Or can you?
A Deliberate Whisper in the Machine
We test our code, our deployments, and our failovers. Why do we so rarely test the very mechanism meant to tell us when the rest of it is broken? The answer is often the perceived risk of a false alarm. No one wants to be the person who triggered a panic over a test. The solution isn't to avoid testing, but to design a test that is, by its nature, unmistakable. I call this technique ‘The Scribe’s Pencil’—a deliberate, intentional mark made in the system’s logbook, designed to be noticed only by those looking for it.
Here is the concrete how-to. First, you need a dedicated, unique log message. It should be something that would never occur under normal operation. Don’t use ‘TEST’ or ‘DEBUG’—these are too common and might be ignored by alerting rules designed to filter out noise. Instead, create a phrase with a specific, unique structure. My current standard is a string like: ‘Scribe’s Pencil: Verification marker for [System Name], timestamp [ISO 8601 timestamp]’.
Once you have your unique marker, you trigger it. Don’t write it directly to a log file from your laptop. That only tests the local filesystem. The goal is to test the entire pipeline: the application’s logging library, the log shipper (like Fluentd or Filebeat), the central aggregation system (like Loki or Elasticsearch), and finally, the alerting rule in PagerDuty, OpsGenie, or your platform of choice. Therefore, the marker must be generated by the actual application, in its real environment. Create a single, obscure API endpoint that, when called, logs this precise message at the INFO level. Secure it fiercely, perhaps with a firewall rule that only allows access from a bastion host or a secret, long-lived token.
Now, the critical part: your alerting rule. You will have a rule that fires on errors, right? Create a separate, parallel rule that looks *only* for this exact message string. Its action should not be to page anyone. Instead, it should send a notification to a low-stakes channel—a dedicated ‘alerts-test’ Slack channel, or a specific email folder. The rule’s runbook is simple: if you see this message, the alerting pipeline is healthy. If you run the test and the message doesn’t arrive within a predictable timeframe (say, five minutes), you know there’s a breakage somewhere in the chain.
This small, almost ceremonial act transforms your alerting system from a black box of faith into a verifiable component. It’s the equivalent of a watchman checking their watch against the town clock. It’s boring, it’s simple, and it instills a profound confidence. Because reliability isn’t just about the servers staying up; it’s about knowing, for certain, that someone will whisper in your ear the moment they start to falter.
Notes & further reading
A few pages I came back to while writing this: