1, Spring Cloud Stream
1. What is it
- Shielding the differences between the underlying message middleware, reducing the switching cost and unifying the programming model of message;
- Officially, Spring Cloud Stream is a framework for building message driven microservices.
- The application interacts with the binder object in Spring Cloud Stream through inputs or outputs. We configure binding, and the binder object of Spring Cloud Stream is responsible for interacting with the message middleware. Therefore, we only need to figure out how to interact with Spring Cloud Stream to facilitate the use of message driven methods.
- By using Spring Integration to connect the message broker middleware to realize message event driven. Spring Cloud Stream provides personalized automatic configuration implementation for some vendors' message middleware products, citing three core concepts: publish / subscribe, consumption group and message partition.
- Currently, only RabbitMQ and Kafka are supported.
Spring Cloud Stream It is a highly scalable event driven microservice framework for building connection with shared messaging system. The framework provides a flexible programming model, which is based on established and familiar Spring idioms and best practices, including three core concepts of publish / subscribe, consumer group and message partition supporting persistence.
Spring Cloud Stream Reference Documentation
Spring Cloud Stream Chinese instruction manual
2. Design idea
2.1 standard MQ
- Information content is transmitted between producers / consumers through message media
- Messages must go through a specific message channel
- How are messages in the message channel consumed and who is responsible for receiving and processing them (SubscribableChannel, a sub interface of the message channel, is subscribed by the MessageHandler message processor)
2.2 Cloud Stream
- For example, we use RabbitMQ and Kafka. Due to the different architectures of the two message middleware, RabbitMQ has exchange and Kafka has Topic and Partitions;
- The differences of these middleware lead to some problems in our actual project development. If we use one of the two message queues and the business requirements behind it, I want to migrate to another message queue. At this time, it is undoubtedly disastrous. A lot of things have to be pushed down and redone because it is coupled with our system, At this time, Spring Cloud Stream provides us with a decoupling method.
- Why can Stream unify the underlying differences?
- Without the concept of binder, when our Spring Boot application needs to directly interact with message oriented middleware, there will be great differences in the implementation details due to the different original intentions of each message oriented middleware;
- By defining the binder as the middle layer, the isolation between application and message middleware details is perfectly realized. By exposing the unified Channel channel to the application, the application does not need to consider a variety of different message middleware implementations.
- By defining Binder as the middle layer, the isolation between application and message middleware details is realized.
- Binder
- The further encapsulation of message oriented middleware by Stream can make the code layer insensitive to the middleware, and even dynamically switch the middleware (Rabbitmq to Kafka), which makes the development of micro services highly decoupled, and services can pay more attention to their own business processes;
- Binder can generate binding. Binding is used to bind producers and consumers of message containers. It has two types: INPUT and OUTPUT. INPUT corresponds to consumers and OUTPUT corresponds to producers.
2.3 the message communication mode in stream follows the publish / subscribe mode
- Broadcast Topic
- In RabbitMQ, it is exchange (switch)
- In Kakfa, it is topic
3. Spring Cloud Stream standard process routine
3.1 Binder
It is convenient to connect middleware to shield differences;
3.2 Channel
Channel is an abstraction of Queue. In the message communication system, it is the medium for storage and forwarding. The Queue is configured through channel;
3.3 Source and Sink
Simply understood, the reference object is the Spring Cloud Stream itself. Publishing messages from the Stream is output, and receiving messages is input.
4. Coding API and common annotations
form | explain |
---|---|
Middleware | Middleware. At present, it only supports RabbitMQ and Kafaka |
Binder | Binder is the encapsulation between application and message middleware. At present, RabbitMQ and Kafaka's binder are implemented. Binder can easily connect middleware and dynamically change message types (corresponding to Kafaka's Topic and RabbitMQ's Exchange). These can be realized through configuration files |
@Input | The annotation identifies the input channel through which messages received enter the application |
@Output | Annotations identify the output channel through which published messages leave the application |
@StreamListener | Listening queue is used for receiving messages from the consumer's queue |
@EnableBinding | It refers to the binding of channel and exchange |
5. Message driven producers
5.1 create a new module cloud stream rabbitmq provider8801
5.2 POM
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <parent> <artifactId>cloud-2020</artifactId> <groupId>com.qs.springcloud</groupId> <version>1.0-SNAPSHOT</version> </parent> <modelVersion>4.0.0</modelVersion> <artifactId>cloud-stream-rabbitmq-provider8801</artifactId> <dependencies> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId> </dependency> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-stream-rabbit</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> </dependency> <!--Hot deployment--> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-devtools</artifactId> <scope>runtime</scope> <optional>true</optional> </dependency> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <optional>true</optional> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> </dependencies> </project>
5.3 YML
server: port: 8801 spring: application: name: cloud-stream-provider cloud: stream: # Configure the service information of rabbitmq to bind here binders: # Represents the name of the definition, which is used for integration with binding defaultRabbit: # Message component type type: rabbit # Set the environment configuration related to rabbitmq environment: spring: rabbitmq: host: 192.168.137.122 port: 5672 username: guest password: guest # Integration of services bindings: # This name is the name of a channel output: # Represents the Exchange name definition to use destination: studyExchange # Set the message type, json this time, and "text/plain" for text content-type: application/json # Set the specific settings of the message service to be bound binder: defaultRabbit eureka: # Configure Eureka registration on the client client: service-url: defaultZone: http://localhost:7001/eureka instance: # Set the heartbeat interval (30 seconds by default) lease-renewal-interval-in-seconds: 2 # If the interval of 5 seconds is exceeded now (the default is 90 seconds) lease-expiration-duration-in-seconds: 5 # Displays the host name in the information list instance-id: send-8801.com # The access path becomes an IP address prefer-ip-address: true
5.4 main startup
@SpringBootApplication public class StreamMQMain8801 { public static void main(String[] args) { SpringApplication.run(StreamMQMain8801.class, args); } }
5.5 IMessageProvider
public interface IMessageProvider { String send(); }
// It can be understood as the definition of a message sending pipeline @EnableBinding(Source.class) public class MessageProviderImpl implements IMessageProvider { // Message sending pipeline @Resource private MessageChannel output; @Override public String send() { String serial = UUID.randomUUID().toString(); // Create and send messages this.output.send(MessageBuilder.withPayload(serial).build()); System.out.println("serial: " + serial); return serial; } }
5.6 SendMessageController
@RestController public class SendMessageController { @Resource private IMessageProvider messageProvider; @GetMapping(value = "/sendMessage") public String sendMessage() { return messageProvider.send(); } }
5.7 testing
http://192.168.137.155:15672/#/
http://localhost:8801/sendMessage
6. Message driven consumers
6.1 create a new module cloud stream rabbitmq consumer 8802
6.2 POM
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <parent> <artifactId>cloud-2020</artifactId> <groupId>com.qs.springcloud</groupId> <version>1.0-SNAPSHOT</version> </parent> <modelVersion>4.0.0</modelVersion> <artifactId>cloud-stream-rabbitmq-consumer8802</artifactId> <dependencies> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId> </dependency> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-stream-rabbit</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> </dependency> <!--Hot deployment--> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-devtools</artifactId> <scope>runtime</scope> <optional>true</optional> </dependency> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <optional>true</optional> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> </dependencies> </project>
6.3 YML
server: port: 8802 spring: application: name: cloud-stream-consumer cloud: stream: # Configure the service information of rabbitmq to bind here binders: # Represents the name of the definition, which is used for binding integration defaultRabbit: # Message component type type: rabbit # Set the environment configuration related to rabbitmq environment: spring: rabbitmq: host: 192.168.137.122 port: 5672 username: guest password: guest # Integration of services bindings: # This name is the name of a channel input: # Represents the Exchange name definition to use destination: studyExchange # Set the message type. This time, it is an object json. If it is text, set "text/plain" content-type: application/json # Set the specific settings of the message service to be bound binder: defaultRabbit eureka: client: # Configure Eureka registration on the client service-url: defaultZone: http://localhost:7001/eureka instance: # Set the heartbeat interval (30 seconds by default) lease-renewal-interval-in-seconds: 2 # If the interval of 5 seconds is exceeded now (the default is 90 seconds) lease-expiration-duration-in-seconds: 5 # Displays the host name in the information list instance-id: receive-8802.com # The access path becomes an IP address prefer-ip-address: true
6.4 main startup
@SpringBootApplication public class StreamMQMain8802 { public static void main(String[] args) { SpringApplication.run(StreamMQMain8802.class, args); } }
6.5 ReceiveMessageListener
@Component @EnableBinding(Sink.class) public class ReceiveMessageListener { @Value("${server.port}") private String serverPort; @StreamListener(Sink.INPUT) public void input(Message<String> message) { System.out.println(serverPort + " Received message: " + message.getPayload()); } }
6.6 testing
http://localhost:8801/sendMessage
7. Consumer groups and persistence
7.1 message repeat consumption
- For example, in the following scenario, when we deploy the order system in a cluster, we will get the order information from RabbitMQ. If an order is obtained by two services at the same time, it will cause data errors. We have to avoid this situation. At this time, we can use the message grouping in the Stream to solve the problem.
- Note that multiple consumers in the same group in the Stream are competitive, which can ensure that messages will only be consumed once by one of the applications. Different groups can be fully consumed (repeated consumption). There will be competition in the same group, and only one of them can be consumed.
- In order to achieve high availability and load balancing, distributed microservice applications actually deploy multiple instances;
- When a producer sends a message to a specific microservice, it only wants to be consumed once. According to the above example of starting two applications, although they belong to the same application, the message is consumed twice. To solve this problem, the concept of consumption group is provided in Spring Cloud Stream.
- grouping
# Different groups # 8002 group: groupA # 8003 group: groupB # Same group # 8002 group: groupQS # 8003 group: groupQS
For multiple micro service instances in the same group, only one will get the message at a time;
7.2 message persistence
- Stop 8802 and 8803, and remove groupQS from 8802
- 8801 sends 4 messages
- Restart 8802, no message received
- Restart 8803 and receive 4 messages
- 8801 sends a message
- 8802 and 8803 receive messages at the same time