The Humble Gatekeeper: A Practical Guide to Fail-Closed with iptables

There is a quiet, almost monastic comfort in a system that knows how to protect itself when you are not there to hold its hand. We spend so much time engineering for graceful failure, for the soft degradation of services, that we sometimes forget the profound utility of the hard stop. For the small, critical service running on a lone server—the internal tool, the archival node, the single-point-of-truth API—there is a virtue in a rule that says, simply, "if I cannot verify my own integrity, then no one shall pass." This is the philosophy of fail-closed, and we can enact it with one of the most boringly reliable tools in the box: iptables.

The Principle of the Sealed Vault

The logic is straightforward but often overlooked. Many services bind to all interfaces (0.0.0.0) for convenience, relying on other layers for security. But what if the service itself crashes or is stopped for maintenance? The port is left open, an unguarded door. A fail-closed approach ensures that door is not just closed but bricked over unless the service itself is alive and explicitly tells the system to open it. The service becomes the sole keeper of its own key.

We achieve this by using a small script, triggered by the service's own process manager (like systemd or supervisord), that manipulates the host's firewall. The default policy for our specific port—say, 9000—is set to DROP all packets. The only thing that can create an exception is the service itself, once it has successfully started and is ready to accept connections.

The Practical Incantation

Here is the concrete technique. First, we set the default rule for our port to deny everything, a one-time command run at boot or as part of a base configuration:

iptables -A INPUT -p tcp --dport 9000 -j DROP

Now, the port is sealed. The next step is to create a script, perhaps /usr/local/bin/open-the-gate, that the service runs as part of its post-start phase. Its contents are simple but powerful:

#!/bin/bash
# Insert our ACCEPT rule at the top of the INPUT chain, so it takes precedence over the DROP.
iptables -I INPUT 1 -p tcp --dport 9000 -j ACCEPT

Conversely, we create a companion script, /usr/local/bin/close-the-gate, for the pre-stop phase:

#!/bin/bash
# Delete our specific ACCEPT rule, leaving the default DROP in place.
iptables -D INPUT -p tcp --dport 9000 -j ACCEPT

Finally, we integrate these into the service's unit file. For a systemd service, this means adding:

[Service]
ExecStartPost=/usr/local/bin/open-the-gate
ExecStopPre=/usr/local/bin/close-the-gate

The effect is beautiful in its reliability. The moment the service starts, it punches a single, precise hole in the firewall. The moment it stops, whether gracefully or due to a crash, that hole is patched. The system reverts to its default state of silent, impenetrable security. It is a form of self-reliance that requires no external monitoring, no complicated health checks—just the simple, mechanical click of a latch being thrown from the inside.

Notes & further reading

A few pages I came back to while writing this: