A variety of ways of distributed system from the perspective of message middleware






As an important member of the distributed system, message middleware has many solutions for major companies and open source. At present, the mainstream open source solutions include RabbitMQ, RocketMQ, Kafka, ActiveMQ, etc. News is simple and difficult to say. The simplicity lies in the ease of use, the simplicity of access, the ability of asynchronous operation to decouple dependencies between systems, and the ability to retroactively retry after failure. The difficulty lies in the design of a set of message mechanism that can support the business and provide a highly available architecture to solve a series of problems such as message storage, message retry, load balancing of message queue, etc. However, the difficulty does not mean that there is no method or "routine". Familiarize yourself with the principle and implementation. If you look at the source code of several frameworks more, you will find out some commonalities.

It is necessary to master the principle and working mechanism of message framework. As for the more used RocketMQ, the design and implementation of the message engine. Alibaba's message engine has developed from Notify to Napoli, and then to MetaQ. Now it is very mature. You can probably see this evolution process in the code of different departments. The current Apache RocketMQ is that Ali donated the MetaQ project to the Apache foundation, while the name of the MetaQ is still used internally.

First, several basic concepts related to messages are explained.

  • A Topic must be established for each message queue.
  • Messages can be grouped, and each message queue requires at least one producer and one Consumer. Producer sends message and Consumer receives consumption message.
  • Each consumer and producer will submit an ID in batches.


RocketMQ system architecture




Next, let's look at RocketMQ's architecture, as shown in the figure, and briefly describe several roles and functions.  

  • NameServer
    • NameServer is the registry of message topics, which is used to discover and manage message producers, consumers, and routing relationships.
  • Broker
    • The relay station of message store and forward uses queue mechanism to manage data store. Multiple message data will be stored in the Broker for fault tolerance. The Master/Slave architecture ensures high availability of the system. A single or multiple masters can be deployed in the Broker. In the scenario of a single Master, after the Master is hung up, the new messages generated by the Producer cannot be consumed, but the messages that have been sent to the Broker can still be consumed by the Consumer due to the existence of the Slave node; if multiple masters are deployed, the system can operate normally.
    • In addition, the Master and Slave in the Broker are not determined by election mechanism in the Zookeeper cluster, but are fixed configurations, which is also the reason why multiple masters need to be deployed in high availability scenarios.
    • After the producer sends the message to the Broker, the Broker will write the message to the local CommitLog file and save the message.
  • Producer
    • The producer will establish a long link with a node in the NameServer cluster, regularly obtain Topic routing information from NamerServeri, and establish a heartbeat with the Broker.
  • Consumer
    • Consumers need to give producers a clear response to the success of consumption, then MetaQ will think that consumption is successful, otherwise it will fail. After failure, RocketMQ will send the message back to Broker and retry within the specified delay time. When the retry reaches a certain number of times (16 times by default), MetaQ thinks that the message cannot be consumed and the message will be delivered to dead letter queue.


Is this architecture familiar? It seems that the architecture of some of the distributed systems I have been exposed to is similar to this one. Even if the roles in the block diagram are changed slightly, it can become the introduction of another framework, such as Dubbo/Redis.

And in the RocketMQ architecture design, the problems to be solved can also be bypassed with other distributed frameworks. Master/Slave mechanism and natural read-write separation are typical solutions of distributed high availability system.

load balancing

Load balancing is another important problem that message framework needs to solve. When the producer produces a large number of messages and the consumer has more than one or more machines, it is necessary to balance the load so that the messages are consumed equally by the consumer. At present, many load balancing algorithms are used in RocketMQ. There are mainly the following: static configuration is too simple to directly configure the queues for consumers to consume, so it is ignored directly.

  1. Average method
  2. Circular queue method
  3. Consistent Hash algorithm
  4. Machine Room algorithm
  5. Static configuration

Take a look at the source code. RocketMQ implements all the above load balancing algorithms internally, and defines an interface AllocateMessageQueueStrategy, which adopts the policy mode. Each load balancing algorithm relies on the implementation of this interface. During operation, an instance of this interface will be obtained, so as to dynamically determine which load balancing algorithm is used.

 1 public interface AllocateMessageQueueStrategy {
 3     /**
 4      * Allocating by consumer id
 5      *
 6      * @param consumerGroup current consumer group
 7      * @param currentCID current consumer id
 8      * @param mqAll message queue set in current topic
 9      * @param cidAll consumer set in current consumer group
10      * @return The allocate result of given strategy
11      */
12     List<MessageQueue> allocate(
13         final String consumerGroup,
14         final String currentCID,
15         final List<MessageQueue> mqAll,
16         final List<String> cidAll
17     );
19     /**
20      * Algorithm name
21      *
22      * @return The strategy name
23      */
24     String getName();
25 }



1. Average method

As the name implies, it is to calculate the average number of consumption queues that a single consumer should bear according to the number of message queues and consumers, and then allocate the message queues to the specified consumers according to the ID of consumers and the way of module. The specific code can be found on Github. The core algorithm code of interception is as follows. mqAll is the structure of message queue, a List of MessageQueue, and cidAll is a List of consumer ID, as well as a List. When mqAll and cidAll are fixed or changed, the current consumer node will get messages from the queue. For example, when the average size is greater than 1, there will be more than one message queue on each consumer, and the ID of the queue allocated to each consumer is continuous.


 1     int index = cidAll.indexOf(currentCID);
 2         int mod = mqAll.size() % cidAll.size();
 3         int averageSize =
 4             mqAll.size() <= cidAll.size() ? 1 : (mod > 0 && index < mod ? mqAll.size() / cidAll.size()
 5                 + 1 : mqAll.size() / cidAll.size());
 6         int startIndex = (mod > 0 && index < mod) ? index * averageSize : index * averageSize + mod;
 7         int range = Math.min(averageSize, mqAll.size() - startIndex);
 8         for (int i = 0; i < range; i++) {
 9             result.add(mqAll.get((startIndex + i) % mqAll.size()));
10         }
11         return result;


2. Ring average method

This algorithm is simpler. First, get the index of the current consumer in the whole list, and get the message queue that the current consumer should deal with directly by using the method of redundancy. Note that the size of mqAll and cidAll can be arbitrary.

  • When ciAll.size() == mqAll.size(), the algorithm is similar to hashtable.
  • When ciAll.size () > mqAll.size () so many consumers can't get the queue of consumption, only some consumers can get the message queue and execute it, which is equivalent to using some consumers to meet the demand without extra cost due to the small number of queues when consumers have sufficient resources.
  • When ciAll.size () < mqAll.size () so that the number of queues that need to be loaded on each consumer exceeds 1, and different from the way of direct average, the consumption queues allocated on each consumer are not continuous, but have a certain step interval.
1         int index = cidAll.indexOf(currentCID);
2         for (int i = index; i < mqAll.size(); i++) {
3             if (i % cidAll.size() == index) {
4                 result.add(mqAll.get(i));
5             }
6         }
7         return result;


3. Consistent Hash algorithm

Cycle all the queues that need to be consumed, calculate the nearest node to process the current queue according to the hash value after the queue toString, and assign it to the node. The routeNode method is a little more complex. If you have time, it's recommended to look at it in detail. Here we just talk about the function.

 1      Collection<ClientNode> cidNodes = new ArrayList<ClientNode>();
 2         for (String cid : cidAll) {
 3             cidNodes.add(new ClientNode(cid));
 4         }
 6         final ConsistentHashRouter<ClientNode> router; //for building hash ring
 7         if (customHashFunction != null) {
 8             router = new ConsistentHashRouter<ClientNode>(cidNodes, virtualNodeCnt, customHashFunction);
 9         } else {
10             router = new ConsistentHashRouter<ClientNode>(cidNodes, virtualNodeCnt);
11         }
13         List<MessageQueue> results = new ArrayList<MessageQueue>();
14         for (MessageQueue mq : mqAll) {
15             ClientNode clientNode = router.routeNode(mq.toString());
16             if (clientNode != null && currentCID.equals(clientNode.getKey())) {
17                 results.add(mq);
18             }
19         }
21         return results;



4. Machine Room algorithm

Hash algorithm based on computer room. This name seems to be a hoax. In fact, it is the same as the above common algorithm for redundancy. There are only more configurations and filters. In order to make this clear, paste the source code completely. You can see that there is an additional member consumeridcs in the implementation class of this algorithm. This is a collection of consumer id. According to certain conventions, broker is named in advance. For example us@metaq4 , and then configure different consumeridcs for different clusters, so as to realize the ability of different machine rooms to handle different message queues.

 1 /*
 2  * Licensed to the Apache Software Foundation (ASF) under one or more
 3  * contributor license agreements.  See the NOTICE file distributed with
 4  * this work for additional information regarding copyright ownership.
 5  * The ASF licenses this file to You under the Apache License, Version 2.0
 6  * (the "License"); you may not use this file except in compliance with
 7  * the License.  You may obtain a copy of the License at
 8  *
 9  *     http://www.apache.org/licenses/LICENSE-2.0
10  *
11  * Unless required by applicable law or agreed to in writing, software
12  * distributed under the License is distributed on an "AS IS" BASIS,
13  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14  * See the License for the specific language governing permissions and
15  * limitations under the License.
16  */
17 package com.aliyun.openservices.shade.com.alibaba.rocketmq.client.consumer.rebalance;
19 import java.util.ArrayList;
20 import java.util.List;
21 import java.util.Set;
22 import com.aliyun.openservices.shade.com.alibaba.rocketmq.client.consumer.AllocateMessageQueueStrategy;
23 import com.aliyun.openservices.shade.com.alibaba.rocketmq.common.message.MessageQueue;
25 /**
26  * Computer room Hashing queue algorithm, such as Alipay logic room
27  */
28 public class AllocateMessageQueueByMachineRoom implements AllocateMessageQueueStrategy {
29     private Set<String> consumeridcs;
31     @Override
32     public List<MessageQueue> allocate(String consumerGroup, String currentCID, List<MessageQueue> mqAll,
33         List<String> cidAll) {
34         List<MessageQueue> result = new ArrayList<MessageQueue>();
35         int currentIndex = cidAll.indexOf(currentCID);
36         if (currentIndex < 0) {
37             return result;
38         }
39         List<MessageQueue> premqAll = new ArrayList<MessageQueue>();
40         for (MessageQueue mq : mqAll) {
41             String[] temp = mq.getBrokerName().split("@");
42             if (temp.length == 2 && consumeridcs.contains(temp[0])) {
43                 premqAll.add(mq);
44             }
45         }
47         int mod = premqAll.size() / cidAll.size();
48         int rem = premqAll.size() % cidAll.size();
49         int startIndex = mod * currentIndex;
50         int endIndex = startIndex + mod;
51         for (int i = startIndex; i < endIndex; i++) {
52             result.add(mqAll.get(i));
53         }
54         if (rem > currentIndex) {
55             result.add(premqAll.get(currentIndex + mod * cidAll.size()));
56         }
57         return result;
58     }
60     @Override
61     public String getName() {
62         return "MACHINE_ROOM";
63     }
65     public Set<String> getConsumeridcs() {
66         return consumeridcs;
67     }
69     public void setConsumeridcs(Set<String> consumeridcs) {
70         this.consumeridcs = consumeridcs;
71     }
72 }


In recent years, due to the expansion and investment of Alibaba's overseas business, middleware such as RocketMQ supports more common overseas business scenarios. Typical scenarios include cross cell consumption and message routing. Cross cell consumption is easy to implement, that is, adding a configuration in consumer to specify the source cell to receive the message, and RocketMQ will internally complete the work of the client pulling the message from the specified cell. Global message routing requires some public resources. The sender of the message can only send the message to a specified unit / machine room, and then route the message to another specified unit. The consumer is deployed in the specified unit. The difference is that one is configured on the client side and the other on the server side.




From the perspective of RocketMQ's design, principle and other distributed frameworks used by individuals, the typical distributed system has to solve the following problems in the design: RocketMQ is all used.

  • Registration and discovery of services. Generally, there will be a unified registration center for management and maintenance.
  • The communication between the service provider and the user can be asynchronous or synchronous. For example, dubbo service synchronizes the service, and the message type is asynchronous communication.
  • HA - high availability architecture. Eight character decision - "master-slave synchronization, separation of reading and writing". To add another sentence can be "live in a different place".
  • Load balancing. Several typical load balancing algorithms have been listed in the content of this article, and the common ones are just these.

Of course, the routines used in the design of message framework are far more than these, including how to ensure the order of message consumption, communication between consumers and server, and message persistence. Similarly, the distributed cache system also needs to solve these problems. Written here, it is quite difficult to fully understand and design such a framework.

Tags: Apache Java Dubbo RabbitMQ

Posted on Sun, 07 Jun 2020 05:00:58 -0400 by PAZII