The Unwelcome Guest: On the Tyranny of the 'Single Source of Truth'
We speak of it in reverent tones, as if naming a deity. The Single Source of Truth. It is the received wisdom, the foundational axiom of so many of our system designs. We are told to build towards it, to venerate its purity, to eliminate any competing narrative that might dare to challenge its authority. But in our quest for this pristine, unified state, I fear we have invited a tyrant into our systems, one that demands absolute obedience and punishes the messy, practical realities of actual operation.
The promise is alluring: one canonical data store, one perfect representation of state that every service queries and every update modifies. In theory, it eliminates the nightmare of reconciliation. There are no conflicting answers, only the One True Answer. But this is a theory that exists on a whiteboard, not in the humming racks of a data center. In practice, the SSOT becomes a monumental single point of failure, a bottleneck of both performance and understanding. It assumes a level of network perfection and transactional purity that is, frankly, a fantasy. When the network partitions—and it will—the truth becomes fragmented and the entire system grinds to a halt, paralyzed by its own doctrinal purity.
Worse, this model fosters a dangerous intellectual laziness. It encourages us to believe that understanding our system is as simple as querying a database. It dismisses the rich, emergent narrative told by a constellation of logs, caches, service states, and even the fleeting contents of message queues. These are not impurities to be eliminated; they are the vital signs of a living system. The log file from Service A might show a request that the "Truth" claims never arrived. Which do you believe? The dogma says the database is correct. The operator, who has learned humility the hard way, knows that the log often holds the more immediate, and sometimes more honest, story.
Perhaps a more honest, more operational philosophy is not a Single Source, but a Convergent Narrative of Truth. We should design systems where multiple services maintain their own state, their own perspective, and are built to reconcile differences as a natural part of their function. This embraces the reality of distributed systems: truth is often local and eventual. It values resilience and availability over a brittle, absolutist consistency. It acknowledges that the 'truth' of what happened is often a story pieced together from multiple witnesses—the database, the log, the cache, the metric—not the decree of a single oracle. It is a messier story, but it is a truer one, and far more robust for the telling.
Notes & further reading
A few pages I came back to while writing this: