Text taste: shredded fish-flavored meat. Read expected: 10 minutes
1. Description
In the previous article, you described what the dead letter queue in RabbitMQ is, when to use it, and how to use it.I believe that through the previous study, you already know more about the Dead Letter Queue, and the content of this article is closely related to the Dead Letter Queue. If you do not know the Dead Letter Queue, then I suggest you read the previous article first.
In this post, we will continue to introduce the advanced features of RabbitMQ, through which you will gain:
- What is a delay queue
- Delayed queue usage scenario
- TTL in RabbitMQ
- How to use RabbitMQ to implement delayed queues
2. Outline of this Article
Here is the outline of this article:
Before reading this article, you need a simple understanding of RabbitMQ and Dead Letter Queues.
3. What is a delay queue
Delayed queue, first of all, it is a queue, which means the elements inside are ordered, the elements out and into the queue are directional, elements enter from one end and take out from the other.
Secondly, the most important feature of a delay queue is its delay property. Unlike a normal queue, elements in a normal queue are always waiting to be removed and processed earlier, while elements in a delay queue are expected to be removed and processed at a specified time, so elements in a delay queue are time-dependent and usually need to be located.Message or task of the manager.
Simply put, a delayed queue is a queue that holds elements that need to be processed at a specified time.
4. Delayed queue usage scenario
So when does a delay queue need to be used?Consider the following scenarios:
- Orders that are not paid within ten minutes are cancelled automatically.
- Newly created stores automatically send message reminders if no items have been uploaded within ten days.
- If the bill is not paid within a week, it is automatically settled.
- After successful user registration, SMS alerts will be sent if there is no login within three days.
- The user initiates a refund and notifies the relevant operator if it has not been processed within three days.
- After the scheduled meeting, each attendee needs to be notified ten minutes before the scheduled time to attend.
These scenarios are characterized by the need to complete a task at a specified point in time after or before an event, such as an order generation event, checking the status of the order payment after ten minutes, then closing the unpaid order, a store creation event, checking the number of new items on the store ten days later, and notifying the merchant with a new number of 0;A bill generation event occurs, checks the bill payment status, and automatically settles unpaid bills; a new user registration event occurs, three days later checks the activity data of the newly registered user, and then notifies the user without any activity records; a refund event occurs, three days later checks whether the order has been processed, and if it has not been processed, sends a message to the relevant operators; when a scheduled meeting occurs, determine if it is only ten minutes from the start of the meeting and, if so, notify the attendees.
It looks like you're using a timed task, polling for data all the time, checking it every second, taking out the data that needs to be processed, and then processing isn't finished?If you have a small amount of data, you can do so, for example, for the requirement of "automatically settle if the bill is not paid in a week", if the time is not strictly limited, but a relaxed week, then running a timed task every night to check all unpaid bills is also a viable option.However, in the case of large amounts of data and time-sensitive scenarios, such as "Order is closed if not paid in 10 minutes". There may be a lot of unpaid order data in a short period of time, even millions or even millions of levels during the activity. It is obviously not advisable to poll for such a large amount of data, and it is likely that all orders will not be completed in one second.Checking also puts a lot of pressure on the database, can't meet business requirements, and has poor performance.
More importantly, no!Excellent!Ya!
Yes, as a aspiring programmer, you should always strive for a more elegant architecture and code style that is as elegant as writing a poem.[Funny]
At this point, the delay queue can shine, and the above scenario is where the delay queue is used.
Since a delayed queue can solve the task requirements with time attribute in many specific scenarios, how do you construct a delayed queue?Next, this article describes how to use RabbitMQ to implement a delayed queue.
5. TTL in RabbitMQ
Before introducing delayed queues, you need to introduce TTL (Time To Live), an advanced feature of RabbitMQ.
What is TTL?TTL is a property of a message or queue in RabbitMQ, indicating the maximum lifetime of a message or all messages in that queue, in milliseconds.In other words, if a message has TTL attributes set or is queued to set TTL attributes, it will become a "dead letter" if it is not consumed within the time set by the TTL (check the previous article for what dead letters are).If both the TTL of the queue and the TTL of the message are configured, the smaller value will be used.
So, how do you set this TTL value?There are two ways, the first is to set the "x-message-ttl" property of the queue when it is created, as follows:
Map<String, Object> args = new HashMap<String, Object>(); args.put("x-message-ttl", 6000); channel.queueDeclare(queueName, durable, exclusive, autoDelete, args);
All messages delivered to this queue will not survive for more than 6 seconds at most.
Another way is to set TTL for each message with the following code:
AMQP.BasicProperties.Builder builder = new AMQP.BasicProperties.Builder(); builder.expiration("6000"); AMQP.BasicProperties properties = builder.build(); channel.basicPublish(exchangeName, routingKey, mandatory, properties, "msg body".getBytes());
The expiration time of this message is then set to 6s.
However, there is a difference between the two methods. If the TTL property of the queue is set, the message will be discarded once it expires. The second way, the message will not be discarded immediately even if it expires, because the expiration of the message is determined before it is about to be delivered to the consumer. If there is a serious message backlog in the current queue, the expired message may beThey can also survive for a long time.
Also, it is important to note that if TTL is not set, the message will never expire and if TTL is set to 0, the message will be discarded unless it can be delivered directly to the consumer at this time.
6. How to Use RabbitMQ to Realize Delayed Queue
If you set up a dead-letter queue, the TTL is described in the previous article. So far, the two elements of the delay queue using RabbitMQ have been assembled. Next, you just need to reconcile them and add a little seasoning to make the delay queue fresh.
Think about a delayed queue, not just how long you want the message to be processed. TTL just makes the message dead letter after how long it has been delayed. On the other hand, messages that become dead letters are delivered to the dead letter queue so that consumers can only consume the messages in the dead letter queue all the time, because the messages inside are messages that they want to be processed immediately.
The flow direction of the message can be roughly seen from the following figure:
Producer produces a delay message, which is routed to different delay queues using different routingkeys depending on the required delay time. Each queue has different TTL attributes set and is bound to the same dead-letter switch. When the message expires, it will be routed to different dead-letter queues depending on the routingkeys. Consumers only need to listen for the corresponding messages.Dead letter queue can be processed.
Here's the code:
First declare the switches, queues, and their binding relationships:
@Configuration public class RabbitMQConfig { public static final String DELAY_EXCHANGE_NAME = "delay.queue.demo.business.exchange"; public static final String DELAY_QUEUEA_NAME = "delay.queue.demo.business.queuea"; public static final String DELAY_QUEUEB_NAME = "delay.queue.demo.business.queueb"; public static final String DELAY_QUEUEA_ROUTING_KEY = "delay.queue.demo.business.queuea.routingkey"; public static final String DELAY_QUEUEB_ROUTING_KEY = "delay.queue.demo.business.queueb.routingkey"; public static final String DEAD_LETTER_EXCHANGE = "delay.queue.demo.deadletter.exchange"; public static final String DEAD_LETTER_QUEUEA_ROUTING_KEY = "delay.queue.demo.deadletter.delay_10s.routingkey"; public static final String DEAD_LETTER_QUEUEB_ROUTING_KEY = "delay.queue.demo.deadletter.delay_60s.routingkey"; public static final String DEAD_LETTER_QUEUEA_NAME = "delay.queue.demo.deadletter.queuea"; public static final String DEAD_LETTER_QUEUEB_NAME = "delay.queue.demo.deadletter.queueb"; // Declare Delayed Exchange @Bean("delayExchange") public DirectExchange delayExchange(){ return new DirectExchange(DELAY_EXCHANGE_NAME); } // Statement Dead Letter Exchange @Bean("deadLetterExchange") public DirectExchange deadLetterExchange(){ return new DirectExchange(DEAD_LETTER_EXCHANGE); } // Declare Delay Queue A Delay 10s // And bind to the corresponding dead letter switch @Bean("delayQueueA") public Queue delayQueueA(){ Map<String, Object> args = new HashMap<>(2); // x-dead-letter-exchange This declares the current queued bound dead-letter switch args.put("x-dead-letter-exchange", DEAD_LETTER_EXCHANGE); // x-dead-letter-routing-key This declares the dead-letter routing key for the current queue args.put("x-dead-letter-routing-key", DEAD_LETTER_QUEUEA_ROUTING_KEY); // TTL of x-message-ttl declaration queue args.put("x-message-ttl", 6000); return QueueBuilder.durable(DELAY_QUEUEA_NAME).withArguments(args).build(); } // Declare Delay Queue B Delay 60s // And bind to the corresponding dead letter switch @Bean("delayQueueB") public Queue delayQueueB(){ Map<String, Object> args = new HashMap<>(2); // x-dead-letter-exchange This declares the current queued bound dead-letter switch args.put("x-dead-letter-exchange", DEAD_LETTER_EXCHANGE); // x-dead-letter-routing-key This declares the dead-letter routing key for the current queue args.put("x-dead-letter-routing-key", DEAD_LETTER_QUEUEB_ROUTING_KEY); // TTL of x-message-ttl declaration queue args.put("x-message-ttl", 60000); return QueueBuilder.durable(DELAY_QUEUEB_NAME).withArguments(args).build(); } // Declare Dead Letter Queue A to receive messages with a delay of 10 seconds @Bean("deadLetterQueueA") public Queue deadLetterQueueA(){ return new Queue(DEAD_LETTER_QUEUEA_NAME); } // Declare Dead Letter Queue B to receive messages with a 60 s delay @Bean("deadLetterQueueB") public Queue deadLetterQueueB(){ return new Queue(DEAD_LETTER_QUEUEB_NAME); } // Declare Delayed Queue A Binding Relationship @Bean public Binding delayBindingA(@Qualifier("delayQueueA") Queue queue, @Qualifier("delayExchange") DirectExchange exchange){ return BindingBuilder.bind(queue).to(exchange).with(DELAY_QUEUEA_ROUTING_KEY); } // Declare Business Queue B Binding Relationships @Bean public Binding delayBindingB(@Qualifier("delayQueueB") Queue queue, @Qualifier("delayExchange") DirectExchange exchange){ return BindingBuilder.bind(queue).to(exchange).with(DELAY_QUEUEB_ROUTING_KEY); } // Declare Dead Letter Queue A Binding Relationship @Bean public Binding deadLetterBindingA(@Qualifier("deadLetterQueueA") Queue queue, @Qualifier("deadLetterExchange") DirectExchange exchange){ return BindingBuilder.bind(queue).to(exchange).with(DEAD_LETTER_QUEUEA_ROUTING_KEY); } // Declare Dead Letter Queue B Binding Relationship @Bean public Binding deadLetterBindingB(@Qualifier("deadLetterQueueB") Queue queue, @Qualifier("deadLetterExchange") DirectExchange exchange){ return BindingBuilder.bind(queue).to(exchange).with(DEAD_LETTER_QUEUEB_ROUTING_KEY); } }
Next, create two consumers to consume messages from two dead-letter queues:
@Slf4j @Component public class DeadLetterQueueConsumer { @RabbitListener(queues = DEAD_LETTER_QUEUEA_NAME) public void receiveA(Message message, Channel channel) throws IOException { String msg = new String(message.getBody()); log.info("Current time:{},Dead Letter Queue A Received message:{}", new Date().toString(), msg); channel.basicAck(message.getMessageProperties().getDeliveryTag(), false); } @RabbitListener(queues = DEAD_LETTER_QUEUEB_NAME) public void receiveB(Message message, Channel channel) throws IOException { String msg = new String(message.getBody()); log.info("Current time:{},Dead Letter Queue B Received message:{}", new Date().toString(), msg); channel.basicAck(message.getMessageProperties().getDeliveryTag(), false); } }
Then the producer of the message:
@Component public class DelayMessageSender { @Autowired private RabbitTemplate rabbitTemplate; public void sendMsg(String msg, DelayTypeEnum type){ switch (type){ case DELAY_10s: rabbitTemplate.convertAndSend(DELAY_EXCHANGE_NAME, DELAY_QUEUEA_ROUTING_KEY, msg); break; case DELAY_60s: rabbitTemplate.convertAndSend(DELAY_EXCHANGE_NAME, DELAY_QUEUEB_ROUTING_KEY, msg); break; } } }
Next, we expose a web interface to produce messages:
@Slf4j @RequestMapping("rabbitmq") @RestController public class RabbitMQMsgController { @Autowired private DelayMessageSender sender; @RequestMapping("sendmsg") public void sendMsg(String msg, Integer delayType){ log.info("Current time:{},Received the request, msg:{},delayType:{}", new Date(), msg, delayType); sender.sendMsg(msg, Objects.requireNonNull(DelayTypeEnum.getDelayTypeEnumByValue(delayType))); } }
Ready to start!
Open rabbitMQ's Manage Background You can see the switch and queue information we just created:
Next, let's send a few messages. http://localhost:8080/rabbitmq/sendmsg?msg=testMsg1&delayType=1 http://localhost:8080/rabbitmq/sendmsg?msg=testMsg2&delayType=2
The logs are as follows:
2019-07-28 16:02:19.813 INFO 3860 --- [nio-8080-exec-9] c.m.d.controller.RabbitMQMsgController : Current time: Sun Jul 28 16:02:19 CST 2019,Received the request, msg:testMsg1,delayType:1 2019-07-28 16:02:19.815 INFO 3860 --- [nio-8080-exec-9] .l.DirectReplyToMessageListenerContainer : SimpleConsumer [queue=amq.rabbitmq.reply-to, consumerTag=amq.ctag-o-qPpkWIkRm73DIrOIVhig identity=766339] started 2019-07-28 16:02:25.829 INFO 3860 --- [ntContainer#1-1] c.m.d.mq.DeadLetterQueueConsumer: Current time: Sun Jul 2816:02:25 CST 2019, Dead Letter Queue A received the message: testMsg1 2019-07-28 16:02:41.326 INFO 3860 --- [nio-8080-exec-1] c.m.d.controller.RabbitMQMsgController : Current time: Sun Jul 28 16:02:41 CST 2019,Received the request, msg:testMsg2,delayType:2 2019-07-28 16:03:41.329 INFO 3860 --- [ntContainer#0-1] c.m.d.mq.DeadLetterQueueConsumer: Current time: Sun Jul 2816:03:41 CST 2019, Dead Letter Queue B received message: testMsg2
The first message becomes a dead letter message after 6s, then consumed by consumers, and the second message becomes a dead letter message after 60s, then consumed, so that a delay queue of ok ay is created.
However, wait a moment. If you do this, don't you need to add a queue for every new time requirement? There are only 6s and 60s time options here. If you need an hour of post processing, you need to add TTL as an hour queue. If you are booking a conference room and then notifying you in advance of such a scenario, don't you want to add an infinite number of queues to fill upIs there enough demand?
Well, when you think about it, it's not easy.
7. RabbitMQ Delay Queue Optimization
Obviously, a more general solution is needed to meet this requirement, so TTL can only be set in the message properties.Let's try.
Add a delay queue to receive messages with an arbitrary delay, and add a corresponding dead letter queue and routingkey:
@Configuration public class RabbitMQConfig { public static final String DELAY_EXCHANGE_NAME = "delay.queue.demo.business.exchange"; public static final String DELAY_QUEUEC_NAME = "delay.queue.demo.business.queuec"; public static final String DELAY_QUEUEC_ROUTING_KEY = "delay.queue.demo.business.queuec.routingkey"; public static final String DEAD_LETTER_EXCHANGE = "delay.queue.demo.deadletter.exchange"; public static final String DEAD_LETTER_QUEUEC_ROUTING_KEY = "delay.queue.demo.deadletter.delay_anytime.routingkey"; public static final String DEAD_LETTER_QUEUEC_NAME = "delay.queue.demo.deadletter.queuec"; // Declare Delayed Exchange @Bean("delayExchange") public DirectExchange delayExchange(){ return new DirectExchange(DELAY_EXCHANGE_NAME); } // Statement Dead Letter Exchange @Bean("deadLetterExchange") public DirectExchange deadLetterExchange(){ return new DirectExchange(DEAD_LETTER_EXCHANGE); } // Declare delay queue C does not set TTL // And bind to the corresponding dead letter switch @Bean("delayQueueC") public Queue delayQueueC(){ Map<String, Object> args = new HashMap<>(3); // x-dead-letter-exchange This declares the current queued bound dead-letter switch args.put("x-dead-letter-exchange", DEAD_LETTER_EXCHANGE); // x-dead-letter-routing-key This declares the dead-letter routing key for the current queue args.put("x-dead-letter-routing-key", DEAD_LETTER_QUEUEC_ROUTING_KEY); return QueueBuilder.durable(DELAY_QUEUEC_NAME).withArguments(args).build(); } // Declare Dead Letter Queue C to receive messages with any delay in processing @Bean("deadLetterQueueC") public Queue deadLetterQueueC(){ return new Queue(DEAD_LETTER_QUEUEC_NAME); } // Declare Delayed Column C Binding Relationships @Bean public Binding delayBindingC(@Qualifier("delayQueueC") Queue queue, @Qualifier("delayExchange") DirectExchange exchange){ return BindingBuilder.bind(queue).to(exchange).with(DELAY_QUEUEC_ROUTING_KEY); } // Declare Dead Letter Queue C Binding Relationships @Bean public Binding deadLetterBindingC(@Qualifier("deadLetterQueueC") Queue queue, @Qualifier("deadLetterExchange") DirectExchange exchange){ return BindingBuilder.bind(queue).to(exchange).with(DEAD_LETTER_QUEUEC_ROUTING_KEY); } }
Add a consumer to Dead Letter Queue C:
@RabbitListener(queues = DEAD_LETTER_QUEUEC_NAME) public void receiveC(Message message, Channel channel) throws IOException { String msg = new String(message.getBody()); log.info("Current time:{},Dead Letter Queue C Received message:{}", new Date().toString(), msg); channel.basicAck(message.getMessageProperties().getDeliveryTag(), false); }
Start again!Then visit: http://localhost:8080/rabbitmq/delayMsg?msg=testMsg1delayTime=5000 To produce the message, notice that the unit here is milliseconds.
2019-07-28 16:45:07.033 INFO 31468 --- [nio-8080-exec-4] c.m.d.controller.RabbitMQMsgController : Current time: Sun Jul 28 16:45:07 CST 2019,Received the request, msg:testMsg1,delayTime:5000 2019-07-28 16:45:11.694 INFO 31468 --- [nio-8080-exec-5] c.m.d.controller.RabbitMQMsgController : Current time: Sun Jul 28 16:45:11 CST 2019,Received the request, msg:testMsg2,delayTime:5000 2019-07-28 16:45:12.048 INFO 31468 --- [ntContainer#1-1] c.m.d.mq.DeadLetterQueueConsumer: Current time: Sun Jul 2816:45:12 CST 2019, Dead Letter Queue C received message: testMsg1 2019-07-28 16:45:16.709 INFO 31468 --- [ntContainer#1-1] c.m.d.mq.DeadLetterQueueConsumer: Current time: Sun Jul 2816:45:16 CST 2019, Dead Letter Queue C received message: testMsg2
It seems okay, but don't be happy too early. At the beginning, I explained that if you set TTL on the message properties, the message might not "die" on time, because RabbitMQ only checks if the first message is out of date, if it is out of date it drops into the dead letter queue, and if the first message has a long delay, the second oneIf the message has a short delay, the second message will not take precedence.
Try it out:
2019-07-28 16:49:02.957 INFO 31468 --- [nio-8080-exec-8] c.m.d.controller.RabbitMQMsgController : Current time: Sun Jul 28 16:49:02 CST 2019,Received the request, msg:longDelayedMsg,delayTime:20000 2019-07-28 16:49:10.671 INFO 31468 --- [nio-8080-exec-9] c.m.d.controller.RabbitMQMsgController : Current time: Sun Jul 28 16:49:10 CST 2019,Received the request, msg:shortDelayedMsg,delayTime:2000 2019-07-28 16:49:22.969 INFO 31468 --- [ntContainer#1-1] c.m.d.mq.DeadLetterQueueConsumer: Current time: Sun Jul 2816:49:22 CST 2019, Dead Letter Queue C received the message: longDelayedMsg 2019-07-28 16:49:22.970 INFO 31468 --- [ntContainer#1-1] c.m.d.mq.DeadLetterQueueConsumer: Current time: Sun Jul 2816:49:22 CST 2019, Dead Letter Queue C received message: shortDelayedMsg
We first sent a message with a delay of 20 seconds and then a message with a delay of 2 seconds. The result shows that the second message will not "die" until the first message becomes a dead letter.
8. Delay Queue by RabbitMQ Plugin
The problem mentioned above is really a hard injury. If you cannot add TTL to the message granularity and make it die in time at the set TTL time, you cannot design a universal delayed queue.
How can I solve this problem?Don't panic, just install a plug-in: https://www.rabbitmq.com/community-plugins.html , Download rabbitmq_Delayed_Message_The exchange plug-in, and then unzip the plug-in directory that you placed in RabbitMQ.
Next, go to the sbin directory under RabbitMQ's installation directory, execute the following command to make the plug-in work, and restart RabbitMQ.
rabbitmq-plugins enable rabbitmq_delayed_message_exchange
Then, let's declare a few more Bean s:
@Configuration public class DelayedRabbitMQConfig { public static final String DELAYED_QUEUE_NAME = "delay.queue.demo.delay.queue"; public static final String DELAYED_EXCHANGE_NAME = "delay.queue.demo.delay.exchange"; public static final String DELAYED_ROUTING_KEY = "delay.queue.demo.delay.routingkey"; @Bean public Queue immediateQueue() { return new Queue(DELAYED_QUEUE_NAME); } @Bean public CustomExchange customExchange() { Map<String, Object> args = new HashMap<>(); args.put("x-delayed-type", "direct"); return new CustomExchange(DELAYED_EXCHANGE_NAME, "x-delayed-message", true, false, args); } @Bean public Binding bindingNotify(@Qualifier("immediateQueue") Queue queue, @Qualifier("customExchange") CustomExchange customExchange) { return BindingBuilder.bind(queue).to(customExchange).with(DELAYED_ROUTING_KEY).noargs(); } }
Add another entry to the controller layer:
@RequestMapping("delayMsg2") public void delayMsg2(String msg, Integer delayTime) { log.info("Current time:{},Received the request, msg:{},delayTime:{}", new Date(), msg, delayTime); sender.sendDelayMsg(msg, delayTime); }
The message producer's code also needs to be modified:
public void sendDelayMsg(String msg, Integer delayTime) { rabbitTemplate.convertAndSend(DELAYED_EXCHANGE_NAME, DELAYED_ROUTING_KEY, msg, a ->{ a.getMessageProperties().setDelay(delayTime); return a; }); }
Finally, create a consumer:
@RabbitListener(queues = DELAYED_QUEUE_NAME) public void receiveD(Message message, Channel channel) throws IOException { String msg = new String(message.getBody()); log.info("Current time:{},Delayed queue receives message:{}", new Date().toString(), msg); channel.basicAck(message.getMessageProperties().getDeliveryTag(), false); }
Everything is ready to start!Then visit the following links:
http://localhost:8080/rabbitmq/delayMsg2?msg=msg1&delayTime=20000 http://localhost:8080/rabbitmq/delayMsg2?msg=msg2&delayTime=2000
The logs are as follows:
2019-07-28 17:28:13.729 INFO 25804 --- [nio-8080-exec-2] c.m.d.controller.RabbitMQMsgController : Current time: Sun Jul 28 17:28:13 CST 2019,Received the request, msg:msg1,delayTime:20000 2019-07-28 17:28:20.607 INFO 25804 --- [nio-8080-exec-1] c.m.d.controller.RabbitMQMsgController : Current time: Sun Jul 28 17:28:20 CST 2019,Received the request, msg:msg2,delayTime:2000 2019-07-28 17:28:22.624 INFO 25804 --- [ntContainer#1-1] c.m.d.mq.DeadLetterQueueConsumer: Current time: Sun Jul 2817:28:22 CST 2019, Delayed queue received message: msg2 2019-07-28 17:28:33.751 INFO 25804 --- [ntContainer#1-1] c.m.d.mq.DeadLetterQueueConsumer: Current time: Sun Jul 2817:28:33 CST 2019, Delayed queue received message: msg1
The second message was consumed first, as expected.At this point, RabbitMQ's implementation of the delayed queue ends.
9. Summary
Delayed queues are useful in scenarios where delay processing is required. Using RabbitMQ to implement delayed queues can make good use of the characteristics of RabbitMQ, such as reliable message sending, reliable message delivery, dead-letter queues to ensure that messages are consumed at least once and messages that are not processed correctly are not discarded.Additionally, the RabbitMQ cluster is a good solution to single point failures without unavailable delayed queues or message loss due to single node hang-up.
Of course, there are many other options for delayed queues, such as using Java's Delay Queu, Redis's zset, Quartz, or kafka's time wheel. Each has its own characteristics, but like stones, this knowledge is like cards in your hand. The more you know, the more cards you can use, and the more you have problems, the more you can move, so you need a lot of knowledge.Knowledge reserve and experience accumulation can create a better card combination to improve their problem solving ability.
On the other hand, with the passage of time and the growth of experience, more and more people feel that they have limited ability to face the complex and changing business needs alone, and in many ways need the help of others to complete the task well.I also know that there are successive smells, specializations in the field, no arrogance anymore, I feel I can get everything done, and I slowly shift my focus to studying how to work effectively in a team. I believe that a highly coordinated team will always be more valuable than a single person fighting.
It took a weekend to finish this article, and all the code was uploaded to github. Https://github.com/MFrank2016/delayed-queue-demoIf you need to check it yourself, I hope it will help you. If there are any errors, you are welcome to correct them and to leave a message with my public number.
