Application of RabbitMq in low concurrent order processing

Basic concepts of RabbitMQ

  • Message Message. The message is anonymous. It consists of a message header and a message body. The message body is opaque, and the message header is composed of a series of optional attributes, including routing key, priority (relative to other messages), delivery mode (indicating that the message may need persistent storage), etc.

  • Publisher The producer of the message is also a client application that publishes the message to the switch.

  • Exchange A switch that receives messages from producers and routes them to queues in the server.

  • Binding Binding, used for association between message queues and switches. A binding is a routing rule that connects the switch and message queue based on the routing key, so the switch can be understood as a routing table composed of bindings.

  • Queue Message queue, used to save messages until they are sent to consumers. It is the container of the message and the end of the message. A message can be put into one or more queues. The message is always in the queue, waiting for the consumer to connect to the queue and take it away.

  • Connection A network connection, such as a TCP connection.

  • Channel A separate two-way data flow channel in a multiplexing connection. The channel is based on the real TCP connection. AMQP commands are sent out through the channel. No matter whether they are publishing messages, subscribing to queues or receiving messages, these actions are completed through the channel. Because it is very expensive for operating system to establish and destroy TCP, the concept of channel is introduced to reuse a TCP connection.

  • Consumer The consumer of the message, representing a client application that gets the message from the message queue.

  • Virtual Host A virtual host that represents a batch of switches, message queues, and related objects. A virtual host is a separate server domain that shares the same authentication and encryption environment. Each vhost is essentially a mini version of RabbitMQ server, with its own queue, switch, binding and permission mechanism. vhost is the basis of AMQP concept, which must be specified when connecting. The default vhost of RabbitMQ is /.

Exchange type

There are four types of distribution strategies in Exchange: direct, fanout, topic and headers. The headers match the AMQP message's header instead of the routing key. In addition, the headers switch and the direct switch are exactly the same, but their performance is poor. At present, they are almost unavailable, so look directly at the other three types

direct

If the routing key in the message is the same as the binding key in the Binding, the switch will send the message to the corresponding queue. The routing key exactly matches the queue name. If a queue is bound to a switch and requires the routing key to be "dog", only messages marked with routing key as "dog" will be forwarded, but not“ dog.puppy ”, and will not forward“ dog.guard ”Wait. It's a perfectly matched, unicast model.

fanout

Every message sent to a fanout type switch is queued to all bindings. The fanout switch does not process the routing key, but simply binds the queue to the switch. Every message sent to the switch will be forwarded to all queues bound to the switch. Much like a subnet broadcast, each host in the subnet gets a copy of the message. The fanout type forwards messages the fastest.

topic

The topic switch allocates the routing key attribute of the message through pattern matching, and matches the routing key with a pattern. At this time, the queue needs to be bound to a pattern. It splits the string of routing key and binding key into words separated by dots. It also recognizes two wildcards: the symbol "ා" and the symbol "*". #Matches 0 or more words, * does not match more than one word.

Application scenario of message queuing

Message communication

The main function of message queue is to send and receive messages. It has an efficient communication mechanism, so it is very suitable for message communication.

We can develop point-to-point chat system based on message queue or broadcast system to broadcast messages to a large number of receivers.

Asynchronous processing

Generally, the programs we write are executed in sequence (synchronous execution), such as a user registration function. The execution sequence is as follows:

  • 1. Write user registration data.

  • 2. Send registration email.

  • 3. Send SMS notification of successful registration.

  • 4. Update statistics.

According to the above execution sequence, you can return success only after all the steps are executed. In fact, after step 1 is executed successfully, other steps can be executed asynchronously. We can send the following logic to the message queue, and then other programs can execute asynchronously, as shown below:

Using message queuing for asynchronous processing can return results faster, speed up the response of the server, and improve the performance of the server.

Service decoupling

In our system, the communication between application and application is very common. Generally, we call directly between applications, for example, application A calls the interface of application B. at this time, the relationship between applications is strong coupling.

If app B is not available, app A will also be affected.

Message queuing is introduced between application A and application B for service decoupling. If application B is hung up, the use of application A will not be affected.

Flow peak shaving

For the high concurrency system, in the peak time of access, the sudden flow flows like a flood to the application system, especially some high concurrency write operations, will cause the database server to be paralyzed at any time, unable to continue to provide services.

The introduction of message queue can reduce the impact of burst traffic on the application system. The message queue is like a "reservoir" to retain the flood in the upstream and reduce the peak flow into the downstream river, so as to achieve the purpose of reducing the flood disaster.

The most common example in this respect is the seckill system. Generally, the instantaneous flow of seckill activities is very high. If all the flow flows to the seckill system, it will crush the seckill system. By introducing the message queue, it can effectively buffer the sudden flow and achieve the function of "peak cutting and valley filling".

RabbitMQ asynchronous notification business mail information

The complete project code address is: https://gitee.com/jiansin/jjlang-exam/tree/master/exam-demo

rabbitmq set mail configuration class
@Configuration
public class RabbitConfig {

    private static final Logger log= LoggerFactory.getLogger(RabbitConfig.class);

    @Autowired
    private CachingConnectionFactory connectionFactory;



    /**
     * Log queue
     * @return
     */
    @Bean
    public Queue logQueue(){
        return new Queue(DemoConstant.LOG_QUEUE_NAME,true);
    }


    /**
     * Log switch
     * @return
     */
    @Bean
    public DirectExchange logDirectExchange(){
        return new DirectExchange(DemoConstant.LOG_EXCHANGE_NAME,true,false);
    }

    /**
     * Binding queues and switches
     * @return
     */
    @Bean
    public Binding logQueueExchangeBinding(){
        return BindingBuilder.bind(logQueue()).to(logDirectExchange()).with(DemoConstant.LOG_ROUTING_KEY);
    }


    @Bean
    public Jackson2JsonMessageConverter jackson2JsonMessageConverter (){
        return new Jackson2JsonMessageConverter();
    }


    @Bean
    public RabbitTemplate rabbitTemplate(){

        RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
        rabbitTemplate.setMessageConverter(jackson2JsonMessageConverter());
        // If setReturnCallback is triggered, mandatory=true must be set. Otherwise, if Exchange does not find the Queue, it will discard the message without triggering the callback
        rabbitTemplate.setMandatory(true);
        rabbitTemplate.setConfirmCallback(new RabbitTemplate.ConfirmCallback() {
            @Override
            public void confirm(CorrelationData correlationData, boolean ack, String cause) {
                //Can handle
                log.info("Message sent successfully:correlationData({}),ack({}),cause({})",correlationData,ack,cause);
            }
        });

        // Whether messages are routed from Exchange to Queue. Note: This is a failure callback. This method will only be called back if messages fail to be routed from Exchange to Queue
        rabbitTemplate.setReturnCallback(new RabbitTemplate.ReturnCallback() {
            @Override
            public void returnedMessage(Message message, int replyCode, String replyText, String exchange, String routingKey) {
                log.info("Message lost:exchange({}),route({}),replyCode({}),replyText({}),message:{}",exchange,routingKey,replyCode,replyText,message);
            }
        });
        return rabbitTemplate;
    }

}

Constant configuration class
/**
 * @description:
 * @author: lilang
 * @version:
 * @modified By:1170370113@qq.com
 */
public class DemoConstant {
    public static final String LOG_QUEUE_NAME = "rabbitmq-log-queue";
    public static final String LOG_EXCHANGE_NAME = "rabbitmq-log-exchange";
    public static final String LOG_ROUTING_KEY = "rabbtimq-log-routingkey";
}
Message production by mail producer
@Service
public class RabbitMqServiceImpl implements RabbitMqService {

    private static final Logger logger= LoggerFactory.getLogger(RabbitMqServiceImpl.class);

    @Autowired
    private RabbitTemplate rabbitTemplate;


    @Autowired
    private ObjectMapper objectMapper;

    @Override

    public String sendInfo() {

        try {
            UserLog userLog = new UserLog("I am sending log message information","200");
            rabbitTemplate.setMessageConverter(new Jackson2JsonMessageConverter());
            rabbitTemplate.setExchange(DemoConstant.LOG_EXCHANGE_NAME);
            rabbitTemplate.setRoutingKey(DemoConstant.LOG_ROUTING_KEY);
            Message message = MessageBuilder.withBody(objectMapper.writeValueAsBytes(userLog)).setDeliveryMode(MessageDeliveryMode.PERSISTENT).build();
            message.getMessageProperties().setHeader(AbstractJavaTypeMapper.DEFAULT_CONTENT_CLASSID_FIELD_NAME, MessageProperties.CONTENT_TYPE_JSON);
            for (int i =0 ; i< 10000 ;i++){
                rabbitTemplate.convertAndSend(message);
            }
        } catch (JsonProcessingException e) {
            logger.info("Message sending exception",e);
        }

        return "Sent successfully";
    }
}    
Message consumption by mail consumers
@Slf4j
@Component
public class BusinessMessageReceiver {

    private static final Logger logger = LoggerFactory.getLogger(BusinessMessageReceiver.class);


    @Autowired
    private ObjectMapper objectMapper;


    /**
     * Listening to consumer logs
     *
     * @param message
     */
    @RabbitListener(queues = DemoConstant.LOG_QUEUE_NAME)
    public void consumeUserLogQueue(Message message,Channel channel) throws IOException {
        try {
            channel.basicQos(1);
            byte []messageBody = message.getBody();
            UserLog userLog = objectMapper.readValue(messageBody, UserLog.class);
            logger.info("Listen to the consumer user log. Listen to the message: {} ", userLog);
            channel.basicAck(message.getMessageProperties().getDeliveryTag(), false);
            //TODO: record the information of sending mail
        } catch (Exception e) {
            logger.info("Exception in message processing", e);
            channel.basicAck(message.getMessageProperties().getDeliveryTag(), false);
        }
    }
}    

Dead letter queue: DLX, dead letter exchange

Reason for dead letter message
1. Message rejected( basic.reject  / basic.nack )And reQueue=false

2. Message TTL expired

When there is a dead letter in the queue, rabbitMQ will automatically republish the message to the set dead letter exchange, and then it will be routed to another queue. You can listen to the message in the queue and do the corresponding processing.

Dead letter queue application
  • Ensure 100% delivery of messages

  • Processing tasks that need to be delayed, such as order invalidation (or scheduled tasks)

Application of dead letter queue to deal with invalid orders in seckill scenario
  • There are many ways to deal with invalid orders, such as timing tasks, etc., which are handled by mq as follows.

  • The following practical use is easy to suspend the database, and MQ can also be used for peak elimination processing. The following is mainly for processing low and invalid orders.

Constant configuration class:

/**
 * @description:
 * @author: lilang
 * @version:
 * @modified By:1170370113@qq.com
 */
public class DemoConstant {

    //Automatic expiration dead letter queue message model for order overdue and unpaid
    //Set up dead letter queue switch
    public static final String MQ_KILL_ITEM_SUCCESS_KILL_DEAD_EXCHANGE = "kill.item.success.kill.dead.exchange";
    //Set dead letter queue route key
    public static final String MQ_KILL_ITEM_SUCCESS_KILL_DEAD_ROUTING_KEY = "kill.item.success.kill.dead.routing.key";
    //Set dead letter queue name
    public static final String MQ_KILL_ITEM_SUCCESS_KILL_DEAD_REAL_QUEUE="kill.item.success.kill.dead.real.queue";
    //Set normal seckill queue name
    public static final String MQ_KILL_ITEM_SUCCESS_KILL_DEAD_QUEUE = "kill.item.success.kill.dead.queue";
    //Set up normal seckill queue switch
    public static final String MQ_KILL_ITEM_SUCCESS_KILL_DEAD_PROD_EXCHANGE="kill.item.success.kill.dead.prod.exchange";
    //Routing key between normal seckill queue switch and normal seckill queue
    public static final String MQ_KILL_ITEM_SUCCESS_KILL_DEAD_PROD_ROUTING_KEY="kill.item.success.kill.dead.prod.routing.key";

    //Milliseconds as a unit
    public static final String KILL_SUCESS_DEAD_EXPIRE = "30000";


}

rabbitmq queue configuration class:

@Configuration
public class RabbitConfig {

    private static final Logger log= LoggerFactory.getLogger(RabbitConfig.class);

    @Autowired
    private CachingConnectionFactory connectionFactory;


    @Bean
    public Jackson2JsonMessageConverter jackson2JsonMessageConverter (){
        return new Jackson2JsonMessageConverter();
    }


    @Bean
    public RabbitTemplate rabbitTemplate(){

        RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
        rabbitTemplate.setMessageConverter(jackson2JsonMessageConverter());
        // If setReturnCallback is triggered, mandatory=true must be set. Otherwise, if Exchange does not find the Queue, it will discard the message without triggering the callback
        rabbitTemplate.setMandatory(true);
        rabbitTemplate.setConfirmCallback(new RabbitTemplate.ConfirmCallback() {
            @Override
            public void confirm(CorrelationData correlationData, boolean ack, String cause) {
                //Can handle
                log.info("Message sent successfully:correlationData({}),ack({}),cause({})",correlationData,ack,cause);
            }
        });

        // Whether messages are routed from Exchange to Queue. Note: This is a failure callback. This method will only be called back if messages fail to be routed from Exchange to Queue
        rabbitTemplate.setReturnCallback(new RabbitTemplate.ReturnCallback() {
            @Override
            public void returnedMessage(Message message, int replyCode, String replyText, String exchange, String routingKey) {
                log.info("Message lost:exchange({}),route({}),replyCode({}),replyText({}),message:{}",exchange,routingKey,replyCode,replyText,message);
            }
        });
        return rabbitTemplate;
    }


    //Build a dead letter queue message model after seckill successfully - the order is overdue and unpaid

    @Bean
    public Queue successKillDeadQueue(){
        Map<String, Object> argsMap= Maps.newHashMap();
        argsMap.put("x-dead-letter-exchange",DemoConstant.MQ_KILL_ITEM_SUCCESS_KILL_DEAD_EXCHANGE);
        argsMap.put("x-dead-letter-routing-key",DemoConstant.MQ_KILL_ITEM_SUCCESS_KILL_DEAD_ROUTING_KEY);
        return new Queue(DemoConstant.MQ_KILL_ITEM_SUCCESS_KILL_DEAD_QUEUE,true,false,false,argsMap);
    }

    //Basic switch
    @Bean
    public TopicExchange successKillDeadProdExchange(){
        return new TopicExchange(DemoConstant.MQ_KILL_ITEM_SUCCESS_KILL_DEAD_PROD_EXCHANGE,true,false);
    }

    //Create basic switch + basic route - > binding of dead letter queue
    @Bean
    public Binding successKillDeadProdBinding(){
        return BindingBuilder.bind(successKillDeadQueue()).to(successKillDeadProdExchange()).with(DemoConstant.MQ_KILL_ITEM_SUCCESS_KILL_DEAD_PROD_ROUTING_KEY);
    }


    //Dead letter queue
    @Bean
    public Queue successKillRealQueue(){
        return new Queue(DemoConstant.MQ_KILL_ITEM_SUCCESS_KILL_DEAD_REAL_QUEUE,true);
    }

    //Dead letter switch
    @Bean
    public TopicExchange successKillDeadExchange(){
        return new TopicExchange(DemoConstant.MQ_KILL_ITEM_SUCCESS_KILL_DEAD_EXCHANGE,true,false);
    }

    //Dead letter switch + dead letter route - > binding of real queue
    @Bean
    public Binding successKillDeadBinding(){
        return BindingBuilder.bind(successKillRealQueue()).to(successKillDeadExchange()).with(DemoConstant.MQ_KILL_ITEM_SUCCESS_KILL_DEAD_ROUTING_KEY);
    }

}

Control layer:

@Controller
@RequestMapping("/kill")
@Api("High concurrency test control class")
public class KillDemoController {

    @Autowired
    private KillService killService;

    /***
     * Commodity seckill core business logic - for stress testing
     */
    @RequestMapping(value ="/iphone",method = RequestMethod.POST,consumes = MediaType.APPLICATION_JSON_UTF8_VALUE)
    @ResponseBody
    public CommonResult killItemLockByRedisson(@RequestBody @Validated KillDto dto){
        //Distributed lock control based on Redisson
        Boolean res=killService.killItemLockByRedisson(dto.getKillId(),dto.getUserId());
        return CommonResult.success(res);
    }

}

Business control layer:

@Service
public class KillServiceImpl implements KillService {

    private static final Logger logger= LoggerFactory.getLogger(KillService.class);

    private SnowFlake snowFlake=new SnowFlake(2,3);

    @Autowired
    private KillDao killDao;

    @Qualifier("redissonSingle")
    @Autowired
    private RedissonClient redissonClient;

    @Autowired
    private RabbitMqService rabbitMqService;

    @Override
    public Boolean killItemLockByZookeeper(Integer killId, Integer userId)  {
        Boolean result=false;

        InterProcessMutex mutex=new InterProcessMutex(curatorFramework,pathPrefix+killId+userId+"-lock");
        try {
            //Time limited waiting
            if (mutex.acquire(10L,TimeUnit.SECONDS)){

                //Processing of core business logic
                //According to the user id, check whether the order id has been killed
                if (killDao.countByKillUserId(killId,userId) <= 0){
                    //Query the corresponding product information
                    //Get seckill product details
                    ItemKill itemKill=killDao.selectItemById(killId);
                    if (itemKill!=null && 1 == itemKill.getCanKill() && itemKill.getTotal()>0){

                        //Deduct inventory
                        int res=killDao.updateKillItemById(killId);

                        if (res>0){
                            //Yes - generate seckill successful order
                            this.insertKillSuccessInfo(itemKill,userId);

                            result=true;
                        }
                    }else {
                        logger.error("The product has been sold out, thank you!");
                    }
                }else{
                    logger.error("redisson-You've already snapped up the product!");
                }
            }
        }catch (Exception e){
            logger.info("There's something wrong with seckill....",e);
        }finally {
            if (mutex!=null){
                try {
                    mutex.release();
                } catch (Exception e) {
                    logger.info("There's something wrong with seckill....",e);
                }
            }
        }
        return result;
    }		
}
    /**
     * General method - record the order generated after the user's seckill success - and notify the asynchronous email message
     * @param kill
     * @param userId
     * @throws Exception
     */
    private void insertKillSuccessInfo(ItemKill kill, Integer userId) throws Exception{
        //Record the seckill order records generated after the successful flash purchase

        ItemKillSuccess entity=new ItemKillSuccess();
        String orderNo=String.valueOf(snowFlake.nextId());

        entity.setCode(orderNo); //Snowflake algorithm
        entity.setItemId(kill.getItemId());
        entity.setKillId(kill.getId());
        entity.setUserId(userId.toString());
        entity.setStatus(SysConstant.OrderStatus.SuccessNotPayed.getCode().byteValue());
        entity.setCreateTime(DateTime.now());
        int res=killDao.insertSucessKill(entity);

        if (res>0){
            //Enter dead letter queue, used for orders that are still unpaid when "expired" exceeds the specified TTL time
            rabbitMqService.sendKillSuccessOrderExpireMsg(orderNo);
        }
    }

mq send message:

    @Override
    public void sendKillSuccessOrderExpireMsg(String orderCode) {

        try {
            if (StringUtils.isNotBlank(orderCode)){
                KillSuccessUserInfo info=killDao.selectByCode(orderCode);
                if (info!=null){

                    rabbitTemplate.setMessageConverter(new Jackson2JsonMessageConverter());
                    rabbitTemplate.setExchange(DemoConstant.MQ_KILL_ITEM_SUCCESS_KILL_DEAD_PROD_EXCHANGE);
                    rabbitTemplate.setRoutingKey(DemoConstant.MQ_KILL_ITEM_SUCCESS_KILL_DEAD_PROD_ROUTING_KEY);

                    rabbitTemplate.convertAndSend(info, new MessagePostProcessor() {
                        @Override
                        public Message postProcessMessage(Message message) throws AmqpException {
                            MessageProperties mp=message.getMessageProperties();
                            //Message setting: when the expiration time of persistent message reaches, it will enter the dead letter queue to update the order status
                            mp.setDeliveryMode(MessageDeliveryMode.PERSISTENT);
                            mp.setHeader(AbstractJavaTypeMapper.DEFAULT_CONTENT_CLASSID_FIELD_NAME,KillSuccessUserInfo.class);
                            //Set message expiration time. Once a message fails, it will enter the dead letter queue
                            mp.setExpiration(DemoConstant.KILL_SUCESS_DEAD_EXPIRE);
                            return message;
                        }
                    });
                }
            }
        }catch (Exception e){
            logger.error("Generate a rush order after seckill succeeds-Send the information to the dead letter queue, waiting for the order expired for a certain period of time and unpaid-An exception occurred with the message:{}",orderCode,e);
        }

    }

Message consumer:

@Component
public class KillMessageReceiver {

    private static final Logger logger = LoggerFactory.getLogger(BusinessMessageReceiver.class);

    @Autowired
    private KillDao killDao;


    @Autowired
    private ObjectMapper objectMapper;

    /**
     * Time out unpaid after user seckill success - listener
     */

    @RabbitListener(queues = DemoConstant.MQ_KILL_ITEM_SUCCESS_KILL_DEAD_REAL_QUEUE)
    public void consumeExpireOrder(Message message, Channel channel){
        try {
            byte []messageBody = message.getBody();
            KillSuccessUserInfo info = objectMapper.readValue(messageBody, KillSuccessUserInfo.class);
            logger.info("The user fails to pay after the second kill-monitor-receive messages:{}",info);

            if (info!=null){
                ItemKillSuccess entity=killDao.selectByPrimaryKey(info.getCode());
                if (entity!=null && entity.getStatus().intValue()==0){
                    killDao.expireOrder(info.getCode());
                }
            }
            channel.basicAck(message.getMessageProperties().getDeliveryTag(), false);

        }catch (Exception e){
            logger.error("The user fails to pay after the second kill-monitor-Exception occurred:",e);
        }
    }

}

Tags: Programming RabbitMQ Database network Attribute

Posted on Wed, 03 Jun 2020 05:20:51 -0400 by abs0lut