We have nearly reached the end of this Module. We are now ready for a more detailed discussion on the interplay between consistency, availability, and performance in distributed systems.

The CAP theorem (introduced in Module 1) seems to indicate that a distributed system can only have two out of consistency, availability, and partition tolerance. This is of course easy to remember, but such an interpretation has received much criticism, from experts such as Daniel Abadi, Peter Bailis, and Michael Stonebraker. Indeed, like any major field of human inquiry with important practical use, the functional properties of distributed systems are hotly debated and, over time, evolve. (Possibly, without consensus and inconsistently, and in ways that do not allow us to establish ‘who said it first’.)

We follow here the summary of existing criticism proposed primarily by Marc Brooker [1].

First, what are the problems with the CAP theorem interpretation of ‘two out of three’?

1. The definition of what a partition is. In Michael Stonebraker’s analysis, it remains unclear when an analyst, or an automated tool, can declare that the system has been partitioned. Surely, a single packet dropped does not constitute a partition. What about intermitted, quickly resolved failures to communicate? Etc. A similar argument can be made about availability, which in distributed systems is often observable only with much effort or after waiting for periods of time much longer than the typical operation in the system.
2. The notion that one can pick any two properties. Few systems can sacrifice partition tolerance and focus only on consistency and high availability.
3. The notion that consistency is either provided or not. In Marc Brooker’s analysis, read your writes and other consistency models can be offered even during a partition, when the system implements client-side caching. This is, of course, not what I observed in the example related to Overleaf, introduced at the start of this Module.

 

Figure 1. The PACELCA framework.

What are the alternatives? PACELCA! (Figure 1)

Introduced by Daniel Abadi, the PACELCA framework clarifies some of the issues raised by the narrower CAP. As Figure 1 illustrates, PACELCA is an acronym that captures a decision tree: if partition tolerance is needed in the system (P), then, decide how the system can trade-off availability (A) vs. consistency (C); else (E), decide how the system will trade-off performance (as latency, L) vs. either consistency (C) or availability (A).

A variety of distributed systems can be described well and, most importantly, distinguished when placed within the PACELCA framework; relatively, the CAP theorem is weaker. For example, all ACID databases, including BigTable and HBase, are PC/EC – they always focus on consistency.

A second large class, of global-scale databases such as Dynamo, Cassandra, and Riak, are PA/EL – they give up the focus on consistency, meaning some inconsistent state is not considered problematic for these classes of systems, and instead focus on maintaining availability when partitions occur and performance when they do not. This fits applications in social media, such as photo or comment browsing; even if we do not see the latest comments momentarily, as long as the system is available, and possibly responsive, we are satisfied.

In a third important class, PA/EC, systems such as the vanilla MongoDB can give up consistency, staying available during moments of partition, and recovering afterward. This class is very close to how CAP works in practice, and thus carries over its ambiguity about what constitutes a partition.

Last, in Daniel Abadi’s formulation, PC/EL systems like PNUTS focus on consistency during partitions, maintaining it to the highest possible level when not all nodes in the system can communicate to each other; when the system is not partitioned, it focuses on performance.

 

In conclusion: CAP or PACELCA?

In short, use PACELCA to describe the trade-off of consistency, availability, and performance during and after system partitions.

 

References:

[1] Marc Brooker, CAP and PACELC: Thinking More Clearly About Consistency, Jul 2014.

[2] Daniel Abadi (2012) Consistency Tradeoffs in Modern Distributed Database System Design: CAP Is Only Part of the Story. IEEE Computer 45(2).