From Traditional Fault Tolerance to Blockchain. Wenbing Zhao

type. It is used for the receiving process is notify the sending process the receiving/execution order of the message. An ORDER message carries the form <ORDER, [m], rsn>, where [m] is the message identifier consisting of a tuple <sender_id, receiver_ id, seq>.

       ◾ ACK message type. It is used for the sending process (of a regular message) to acknowledge the receipt of the ORDER message. It assumes the form <ACK, [m]>.

       2.3.2.2 Normal Operation of the Message Logging Protocol

The normal operation of the protocol is shown in Figure 2.17.

Figure 2.17 Normal operation of the sender-based logging protocol.

      The protocol operates in three steps for each message:

      1 A REGULAR message, <REGULAR,seq,rsn,m>, is sent from one process, e.g., Pi, to another process, e.g., Pj.

      2 Process Pj determines the receiving/execution order, rsn, of the regular message and informs the determinant information to Pi in an ORDER message <ORDER, [m], rsn>.

      3 Process Pj waits until it has received the corresponding acknowledgment message, <ACK, [m]>, before it sends out any REGULAR message.

The original sender-based message logging protocol [13] was designed for use with unreliable channels. Since we have assumed the use of reliable channels, one might wonder if the third step in the protocol is still necessary. The answer is yes because transport-level reliability does not necessarily lead to application-level reliability, as we have argued in section 2.3.1.2. If a process sends the ordering message to a process and another regular message to a different process, and node on which the process runs subsequently crashes, the ordering message might not be delivered to its intended target successfully while the regular message might.

      Furthermore, in the original sender-based message logging protocol [13] , the regular message and the ordering message must be retransmitted after a timeout before the expected acknowledgment message is received. With the use of reliable channels, such proactive retransmission becomes unnecessary because the only scenario in which a retransmission is necessary is when a process fails, in which case, the retransmission will be triggered by the recovery mechanism (more in section 2.3.2.3).

      The use of a mature reliable communication protocol such as TCP in distributed applications is more desirable because the application developers can focus on the application logic and application-level messaging reliability without worrying about issues such as achieving high throughput and doing congestion control.

      EXAMPLE 2.6

In the example shown in Figure 2.18, the distributed system consists of three processes. Both the seq counter and rsn counter are initialized to be 0, and the message log is empty at each process. Process P₀ first sends a regular message, <REGULAR,0,?,m₀>, to P₁. Upon sending the message, P₀ increments its seq counter to 1 and log the message in its volatile buffer. At this point, the rsn value for the message is unknown, hence it is denoted as a question mark.

      On receiving the regular message <REGULAR,0,?,m₀>, P₁ assigns the current rsn counter value, which is 0, to this message indicating its receiving order, increments its rsn counter to 1, and sends P₀ an ORDER message <ORDER,[m₀],0>. When P₀ receives this ORDER message, it updates the entry in its message log to reflect the ordering number for message m₀, and sends an sc ack message, <ACK,[m₀]>, to P₁.

      Once receiving the ACK message, P₁ is permitted to send a regular message, <REGULAR,0,?,m₁>, to P₂. The handling of the message and the corresponding ORDER and ACK messages are similar to the previous ones.

Figure 2.18 An example normal operation of the sender-based logging protocol.

      Subsequently, P₀ and P₂ send three regular messages m₂, m₃, m₄, nearly concurrently to P₀. P₁ assigns 1 as the rsn value for the first of the three messages (for m₂) and sends an ordering message to P₀, and assigns 2 and 3 for the two back-to-back regular messages (for m₃ and m₄) from P₂. For the two messages from P₂, P₁ can batch the ORDER messages and sends them together to P₂, and P₂ can batch the corresponding the ACK messages to P₁ too. Upon receiving the ACK messages for all three ORDER messages, P₁ sends another regular message containing m₅ with sequence number 1, updates the seq counter to 2, and log the message.

2.3.2.3 Recovery Mechanism.

      On recovering from a failure, a process first restores its state using the latest local checkpoint, and then it must broadcast a request to all other processes in the system to retransmit all their logged messages that were sent to the process.

Because the checkpoint includes its message log, and the regular messages logged and the corresponding ACK messages might not reach their the destination processes due to the process failure, the recovering process retransmit the regular messages or the ack messages based on the following rule:

       ◾ If the entry in the log for a message contains no rsn value, then a REGULAR message is retransmitted because the intended receiving process might not have received this message.

       ◾ If the entry in the log for a message contains a valid rsn value, then an ACK message is sent so that the receiving process can send regular messages.

      When a process receives a regular message, it always sends a corresponding ORDER message in response. There are three scenarios:

       ◾ The message is not a duplicate, in which case, the current rsn counter value is assigned to the message as its receiving order, and the corresponding ORDER message is sent. The process must then wait for the ACK message before it sends any regular message.

       ◾ The message is a duplicate, and the corresponding rsn value is found in its history list, in which case, an ORDER is message is sent and the duplicate message itself is discarded. The process must then wait for the ACK message before it sends any regular message. Note that it is impossible for the process to have received the corresponding ACK message before because otherwise the recovering process must have logged the rsn value for the regular message.

       ◾ The message is a duplicate, and there is no corresponding entry in the history list. In this case, the process must have checkpointed its state after receiving the message and it is no longer needed for recovery. As a result, the process sends an ORDER message with a special constant indicating that the message is no longer needed and the sending processing can safely purge the entry from its message log.

The