1.2.5 Time Synchronization
One way of creating consistency between multiple concurrent data updates would be using time as a canonical ordering. If this would be viable, we could simply mandate that all replicas of the data apply the updates in exactly the order that they happened.
Unfortunately, we cannot rely on global time in distributed systems. Every machine has its own local clock and without any additional measures these clocks are neither synchronized, nor will they run at exactly the same speed, causing clock drift over time. In essence, time is just a special case of global state that we do not have in distributed systems.
While there are approaches for synchronizing Real-Time Clocks such as NTP and Truetime, some systems can tolerate not knowing the exact time. For example, a game using the distributed simulation approach may not care about exactly when each action happened. It only needs to know in which simulation step each action takes place. This reduces the problem to determining the order of actions, instead of the exact time of each action. Logical clocks are a well-known solution for deciding the order of operations. Here we discuss two well-known examples of such clocks: Lamport clocks and vector clocks:
Lamport Clocks do not keep track of real time, but count the number of events on a machine. Sending and receiving messages are counted as events. What else counts as an event does not matter for the algorithm. This yields a monotonically-increasing value on all machines. Lamport clocks are built around the concept of a happened-before relation between events. If a and b are events, then a → b means that event a happened before event b. The following pairs of events have a happened-before relation:
- All events that take place on a single system have a happened-before relation. The event with the smallest timestamp happened first.
- Events that happen on different machines have a happened-before relation if they are connected by the sending and receiving of messages. If a process P1 sends a message to P2, then all events on P1 that happened before the send operation also happened before all events on P2 at the time that P2 receives the message.
Machines add a timestamp (their current Lamport clock value) to every message they send. A receiver compares the attached timestamp with its own clock. If the timestamp is lower than its current clock value it does nothing. But when the timestamp is higher than its current clock value, it sets its own clock to timestamp+1. Doing this guarantees that if a happened before b, than a’s logical clock value is smaller than b’s.
An example of two machines using Lamport clocks is shown in this figure. Two events have a happened-before relation if there is a path from the first to the second event by either moving down along a single process, or following a message to another process. For example, event with timestamp 40 on P2 happened before event with timestamp 0 on P1.
Events that do not have a happened-before relation are defined as concurrent. In the context of logical clocks, concurrent does not mean that the events are executed at the same moment in real time. Instead, it indicates that the system is unaffected by the order of these operations. The system would have arrived at the same state, no matter which event happened first in real time.
Lamport Clocks can be used to create a total order of operations. When a machine sends a request, it attaches its Lamport clock value and sends it to all other machines. Receivers place incoming messages in a queue sorted by clock value, and send an acknowledgment to the sender and all other machines. Machines update their own Lamport clock value based on the value attached to the messages they receive, if necessary. Machines deliver a message (remove it from the queue) if it is at the head of the queue, and has been acknowledged by all other machines. A downside of this approach is its inefficiency. Every operation requires n^2 messages for n machines.
Although Lamport clocks can be used to create a total order, comparing two events with timestamps x and y where x < y cannot tell us if x happened before y. We can only conclude that y did not happen before x. x happened before y, or x and y are concurrent.
In contrast, vector clocks, which are an extension of Lamport clocks, capture causality. Vector clocks allow us to determine for each pair of events which happened before the other, or that they are concurrent, by inspecting their clock values. To do this, machines do not only store their own Lamport clock, but also those of all other machines, yielding a vector of clocks.
Machines update their own Lamport clock using the same rules used for regular Lamport clocks. The other vector entries, the ones corresponding to the other machines, are updated when vector clocks are received from other machines. Every machine appends to every message a copy of its vector clock. When receiving a message, machines update their vector clock by increasing their own counter by one, and taking the pair-wise maximum for the other vector clock values. An example of machines using vector clocks is shown in the following figure:
Deciding the causality for each pair of events is now straightforward. For a pair of events, a and b, a happened before b if a’s vector timestamp is strictly smaller than b’s. If one timestamp is not strictly larger than the other, the events are concurrent. Looking at the figure, we see that the event with timestamp (2, 1, 0) at P1 happened before the event with timestamp (2, 3, 1) at P3, because 2 ≤ 2, 1 < 3, and 0 < 1. The event with timestamp (2, 3, 0) at P2 is concurrent with the event with timestamp (3, 1, 0) at P1 because (3, 1, 0) ≮ (2, 3, 0) and (2, 3, 0) ≮ (3, 1, 0).
Modern Distributed Systems by TU Delft OpenCourseWare is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Based on a work at https://online-learning.tudelft.nl/courses/modern-distributed-systems/