Lightweight RabbitMQ automatic configuration and exception handling component

Development background

  • rabbitmq is used in the project. When a new switch or queue is introduced, exchange, queue and routingKey need to be configured. Repetitive work is cumbersome. Multiple copies need to be configured when the consumer side and the production side are separated, which is easy to make mistakes
  • Messages are lost when sending and consuming messages, so it is impossible to guarantee 100% delivery and 100% consumption of messages
  • Repeated consumption at the consumer end

Function description

  • It supports zero configuration when introducing new queues, and automatically configures message queues and switches without manual configuration to avoid configuration errors
  • Support manual configuration, compatible with old queue configuration, support custom queue name, switch name and switch mode
  • Support to stop and start consumer monitoring
  • Guarantee 100% delivery of messages, support asynchronous retry after sending failure listening
  • Guarantee 100% consumption of messages, support asynchronous retry after consumption failure
  • API is provided to control the processing of sending failure and message failure queues, such as modifying message body, re consuming, re sending, etc
  • Encapsulates the re queuing of simple timed task processing messages
  • Solve the problem of repeated consumption at the consumer end

Use cases

Zero configuration is realized. You don't need to care about the configuration details, only need to concern your own business logic

1) Only the production side is required

@Service("smsMsgQueueService")
public class SmsMsgQueueService extends AbstractMsgQueueService<SMSEntity> {

    @Override
    protected String getMessageDesc() {
        return "Sending SMS";
    }
}

// Caller

@Resource(name = "smsMsgQueueService")
private IMsgQueueService smsMsgQueueService;

public void sendSms(SMSEntity smsEntity) {
    smsService.provide(JsonHelper.serialize(smsEntity));
}

2) Production end + consumer end

Just implement the protected void consumemessage (smsententity smsententity, messageproperties messageproperties)

@Service("smsMsgQueueService")
public class SmsMsgQueueService extends AbstractMsgQueueService<SmsEntity> {
    
    @Override
    protected String getMessageDesc() {
        return "Sending SMS";
    }

    @Override
    protected void consumeMessage(SmsEntity smsEntity, MessageProperties messageProperties) throws Throwable {
        // Message specific processing class
    }
}

How to implement the following method without message header information

protected void consumeMessage(SmsEntity smsEntity) throws Throwable() {}

Usage method

  • Maven dependency:
<dependency>
    <groupId>com.shanwtime</groupId>
    <artifactId>basicmq</artifactId>
    <version>1.0-SNAPSHOT</version>
</dependency>
  • Add package scanning configuration class [com.shawntime.basic.dao]
static final String SCAN_PACKAGE = "com.shawntime.provider.dao.mapper." + DB_TGA + ",com.shawntime.basic.dao";
Resource[] locationResources = resolver.getResources(mapperLocations);
Resource[] mqResources = resolver.getResources("classpath:mapper/basicmq/MessageQueueErrorLogMapper.xml");
List<Resource> resources = new ArrayList<>(locationResources.length + mqResources.length);
resources.addAll(Arrays.asList(locationResources));
resources.addAll(Arrays.asList(mqResources));
sqlSessionFactoryBean.setMapperLocations(resources.stream().toArray(Resource[]::new));
  • Packet scan
@ComponentScan(basePackages = {"com.shawntime.basic")
  • yml file configuration rabbitmq link parameters
spring:
  rabbitmq:
    base:
      host: 127.0.0.1
      port: 5672
      username: admin
      password: 123456
      vhost: test
      isOpenListener: true

Isonlistener: whether to turn on the consumer side, which is off by default

  • To enable the component's own scheduled tasks, you need to configure application.properties
openScheduledTask=true
Modification of old projects

The implementation class of basicmq project has been introduced, and the following two methods need to be rewritten
isAutoRegistry: whether to register automatically
isConfirmCallBack: confirm whether the switch receives messages

@Service
public class DingTalkService extends AbstractMsgQueueService<DingMessage> {

    @Resource(name = "dingTalkTemplate")
    private AmqpTemplate dingTalkTemplate;

    @Value("${spring.rabbitmq.tmApi.queue.dingTalk}")
    private String dingTalkQueue;

    @Value(" ${spring.rabbitmq.tmApi.exchange.dingTalk}")
    private String dingTalkExchange;

    @Override
    protected String getMessageDesc() {
        return "Nail message";
    }

    @Override
    protected void provideMessage(String msgBodyJson) throws Throwable {
        dingTalkTemplate.convertAndSend(dingTalkQueue, msgBodyJson);
    }

    @Override
    protected boolean isAutoRegistry() {
        return false;
    }

    @Override
    protected boolean isConfirmCallBack() {
        return false;
    }
}

code analysis

Production side message 100% delivery
@Override
public void provide(String msgBodyJson, boolean isAsync, Map headMap) {
    try {
        logger.info("provide -> {}", msgBodyJson);
        String correlationDataId = "";
        if (isConfirmCallBack()) {
            MessageData data = getMessageData(msgBodyJson);
            redisClient.hset(Constant.queue_key, data.getId(), JsonHelper.serialize(data), -1);
            correlationDataId = data.getId();
        }
        provideMessage(msgBodyJson, correlationDataId, headMap);
    } catch (Throwable e) {
        exceptionHandle(new MsgQueueBody(BasicOperatorEnum.PROVIDER, msgBodyJson), e, isAsync);
    }
}

private MessageData getMessageData(String msgBodyJson) {
    String id = UUID.randomUUID().toString();
    MessageData data = new MessageData();
    data.setCurrTime(System.currentTimeMillis());
    data.setId(id);
    data.setJsonData(msgBodyJson);
    data.setTypeDesc(getMessageDesc());
    data.setTypeId(getMessageType());
    data.setOriginalId(0);
    data.setBeanName(getSpringBeanName());
    return data;
}

protected void provideMessage(String msgBodyJson,
                                  String correlationDataId,
                                  Map<String, Object> headMap) throws Throwable {
    if (!isAutoRegistry()) {
        provideMessage(msgBodyJson, correlationDataId);
        return;
    }
    String exchangeName = getExchangeName();
    String queueName = getQueueName();
    MessagePostProcessor messagePostProcessor = message -> {
        MessageProperties messageProperties = message.getMessageProperties();
        if (MapExtensions.isNotEmpty(headMap)) {
            headMap.forEach((key, value) -> {
                messageProperties.setHeader(key, value);
            });
        }
        messageProperties.setCorrelationId(correlationDataId.getBytes());
        return message;
    };
    amqpProducer.publishMsg(exchangeName, queueName, msgBodyJson,
            new CorrelationData(correlationDataId), messagePostProcessor);
}
Sender message confirmation
@Component("confirmCallBackListener")
public class ConfirmCallBackListener implements RabbitTemplate.ConfirmCallback {

    @Resource
    private RedisClient redisClient;

    @Override
    public void confirm(CorrelationData correlationData, boolean ack, String cause) {
        String id = correlationData.getId();
        if (StringUtils.isEmpty(id)) {
            return;
        }
        if (ack) {
            redisClient.hdel(Constant.queue_key, id);
        }
    }

}
Send failed message job asynchronous retry
@Override
public void retry() {
    Map<String, String> keyMap = redisClient.hgetAll(Constant.queue_key);
    if (MapExtensions.isEmpty(keyMap)) {
        return;
    }
    keyMap.forEach((id, value) -> {
        MessageData data = JsonHelper.deSerialize(value, MessageData.class);
        if (System.currentTimeMillis() - data.getCurrTime() > 5 * 60 * 1000) {
            MessageQueueErrorRecord log = new MessageQueueErrorRecord();
            log.setBeanName(data.getBeanName());
            log.setErrorDesc("");
            log.setIsRePush(0);
            log.setMsgBody(data.getJsonData());
            log.setTypeDesc(data.getTypeDesc());
            log.setOperatorId(BasicOperatorEnum.PROVIDER.getCode());
            log.setTypeId(data.getTypeId());
            log.setOriginalId(data.getOriginalId());
            msgQueueErrorLogService.save(log);
            redisClient.hdel(Constant.queue_key, id);
        }
    });
}
100% consumption at the consumer end

Processing failure message unified warehousing

@Override
public void consume(String msgBodyJson, int originalId, MessageProperties messageProperties) {
    try {
        logger.info("consume -> {}", msgBodyJson);
        T obj = JsonHelper.deSerialize(msgBodyJson,
                (Class<T>) ((ParameterizedType) getClass().getGenericSuperclass()).getActualTypeArguments()[0]);
        if (messageProperties == null || messageProperties.getCorrelationId() == null) {
            consumeMessage(obj, messageProperties);
            return;
        }
        String correlationId = new String(messageProperties.getCorrelationId(), Charset.defaultCharset());
        if (StringUtils.isEmpty(correlationId)) {
            consumeMessage(obj, messageProperties);
            return;
        }
        if (exchangeType() == ExchangeType.TOPIC || exchangeType() == ExchangeType.FANOUT) {
            consumeMessage(obj, messageProperties);
            return;
        }
        String lockKey = "lock." + correlationId;
        boolean isLock = RedisLockUtil.lock(lockKey, correlationId, 60);
        if (isLock) {
            try {
                consumeMessage(obj, messageProperties);
            } finally {
                RedisLockUtil.unLock(lockKey, correlationId);
            }
        } else {
            new RuntimeException("Repeated consumption");
        }
    } catch (Throwable e) {
        exceptionHandle(new MsgQueueBody(BasicOperatorEnum.CONSUMER, msgBodyJson), e, false, originalId);
    }
}

private void exceptionHandle(MsgQueueBody msg, Throwable throwable, boolean isAsync, int originalId) {
    logger.error(getMessageDesc() + "|" + msg.getMsgQueueBody(), throwable);
    MessageQueueErrorRecord log = new MessageQueueErrorRecord();
    log.setMsgBody(msg.getMsgQueueBody());
    log.setErrorDesc(DBStringUtil.subString(throwable.getMessage(), 1500));
    log.setOperatorId(msg.getBasicOperatorEnum().getCode());
    String beanName = getSpringBeanName();
    log.setBeanName(beanName);
    log.setTypeId(getMessageType());
    log.setTypeDesc(StringUtils.defaultString(getMessageDesc(), ""));
    log.setOriginalId(originalId);
    logger.info("dbLog -> {}", JsonHelper.serialize(log));
    if (isAsync) {
        rabbitProductExecutor.submit(() -> saveLog(log));
    } else {
        saveLog(log);
    }
}
Exception message retry
private void rePush(MessageQueueErrorRecord record) {
    int id = record.getOriginalId();
    if (!IntegerExtensions.isMoreThanZero(record.getOriginalId())) {
        id = record.getId();
    }
    if (!check(id)) {
        return;
    }
    AbstractMsgQueueService messageQueueService =
            (AbstractMsgQueueService) msgQueueFactory.getMsgQueueService(record);
    messageQueueService.consume(record.getMsgBody(), id);
    record.setIsRePush(1);
    msgQueueErrorLogService.update(record);
}

private boolean check(int id) {
    String key = getRedisKey(id);
    Long increment = redisClient.increment(key, 1L, 24 * 60 * 60);
    return increment == null || increment.longValue() <= 3;
}

private String getRedisKey(int id) {
    return "message.queue.consume.limit"
            + "." + LocalDateUtil.format(new Date(), "yyyyMMdd")
            + "#" + id;
}
Configure auto injection monitoring
@PostConstruct
public void autoRegistry() {
    if (!isAutoRegistry()) {
        return;
    }
    DefaultListableBeanFactory beanFactory = (DefaultListableBeanFactory) SpringUtils.getBeanFactory();
    ConnectionFactory connectionFactory = (ConnectionFactory) beanFactory.getBean("basicConnectionFactory");
    RabbitAdmin rabbitAdmin = (RabbitAdmin) beanFactory.getBean("basicRabbitAdmin");
    DynamicConsumer consumer = null;
    try {
        DynamicConsumerContainerFactory fac = DynamicConsumerContainerFactory.builder()
                .exchangeType(exchangeType())
                .exchangeName(getExchangeName())
                .queue(getQueueName())
                .autoDeleted(false)
                .autoAck(true)
                .durable(true)
                .routingKey(getRoutingKey())
                .rabbitAdmin(rabbitAdmin)
                .connectionFactory(connectionFactory).build();
        consumer = new DynamicConsumer(fac, this);
    } catch (Exception e) {
        logger.error("System exception", e);
    }
    customizeDynamicConsumerContainer.put(getQueueName(), consumer);
    if (isOpenListener()) {
        consumer.start();
    }
}
AbstractMsgQueueService abstract class use
send message
/**
* msgBodyJson: json Serialized message body
* isAsync: Asynchronous receipt or not after sending failure
* headMap: Message header carried when sending a message
*/
public void provide(String msgBodyJson)
public void provide(String msgBodyJson, Map headMap)
public void provide(String msgBodyJson, boolean isAsync, Map headMap)
Consumer News
/**
* msgBodyJson: Message body received
* originalId: The primary key id of the original message substituted by retry, which is used to control the number of retries per day of the message
* messageProperties: Message header information
*/
public void consume(String msgBodyJson)
public void consume(String msgBodyJson, int originalId)
public void consume(String msgBodyJson, MessageProperties messageProperties)
public void consume(String msgBodyJson, int originalId, MessageProperties messageProperties)
Find entity bean

If messageType is defined, it will be found by messageType; if not, it will be found by bean name of service

protected int getMessageType()
private String getSpringBeanName()
Custom exchange, queue, exchange type, routingKey

Override the following methods

protected String getQueueName()
protected String getDirectExchangeName()
protected String getTopicExchangeName()
protected String getFanoutExchangeName()
protected ExchangeType exchangeType()
protected String getRoutingKey()

api interface introduction

  • mq/repush/ids: push failure message again according to multiple primary key IDS
  • mq/repush/id: push failure message again according to primary key id
  • MQ / repush / type id s: re push all failed messages of a type according to multiple type IDS
  • mq/repush/typeId: re push all failed messages of a type according to the type id
  • mq/modify/status/id: modify the message weight test status according to the primary key id
  • mq/modify/status/ids: modify message retry status based on multiple primary key IDS
  • mq/modify/status/typeId: modify the message retry status according to the type
  • mq/add/typeId: add a message based on the type id
  • mq/add/beanName: add messages according to beanName
  • mq/retry: production side failure message retry
  • mq/listener/close/typeId: turn off message listening of this type according to the type id
  • mq/listener/close/beanName: turn off message listening of beanName according to beanName
  • mq/listener/start/typeId: start message listening of this type according to the type id
  • mq/listener/start/beanName: start message listening of beanName according to beanName
Create database
  • SQL Server
CREATE TABLE [dbo].[MessageQueueErrorRecord] (
  [id] int  IDENTITY(1,1) NOT NULL,
  [operator_id] tinyint  NOT NULL,
  [type_id] int  NOT NULL,
  [type_desc] varchar(50) COLLATE Chinese_PRC_CI_AS  NOT NULL,
  [bean_name] varchar(200) COLLATE Chinese_PRC_CI_AS  NOT NULL,
  [msg_body] varchar(8000) COLLATE Chinese_PRC_CI_AS  NOT NULL,
  [error_desc] varchar(2000) COLLATE Chinese_PRC_CI_AS  NOT NULL,
  [original_id] int DEFAULT ((0)) NOT NULL,
  [is_re_push] tinyint DEFAULT ((0)) NOT NULL,
  [created_stime] datetime DEFAULT (getdate()) NOT NULL,
  [modified_stime] datetime DEFAULT (getdate()) NOT NULL,
  [is_del] int DEFAULT ((0)) NOT NULL
)
GO

ALTER TABLE [dbo].[MessageQueueErrorRecord] SET (LOCK_ESCALATION = TABLE)
GO

EXEC sp_addextendedproperty
'MS_Description', N'Primary key id',
'SCHEMA', N'dbo',
'TABLE', N'MessageQueueErrorRecord',
'COLUMN', N'id'
GO

EXEC sp_addextendedproperty
'MS_Description', N'Operation type, 1:Consumer, 2: Production',
'SCHEMA', N'dbo',
'TABLE', N'MessageQueueErrorRecord',
'COLUMN', N'operator_id'
GO

EXEC sp_addextendedproperty
'MS_Description', N'Message type id,1001-"Consumer side: original order warehousing",2001-"Production side: original order warehousing",1002-"Consumer end: million agent order push",2002-"Production side: million agent order push",1003-"Consumer end: send SMS",2003-"Production end: add SMS information",1004-"Consumer: order export",2004-"Production side: order export"',
'SCHEMA', N'dbo',
'TABLE', N'MessageQueueErrorRecord',
'COLUMN', N'type_id'
GO

EXEC sp_addextendedproperty
'MS_Description', N'Message type description',
'SCHEMA', N'dbo',
'TABLE', N'MessageQueueErrorRecord',
'COLUMN', N'type_desc'
GO

EXEC sp_addextendedproperty
'MS_Description', N'Message correspondence spring bean Name',
'SCHEMA', N'dbo',
'TABLE', N'MessageQueueErrorRecord',
'COLUMN', N'bean_name'
GO

EXEC sp_addextendedproperty
'MS_Description', N'Message body',
'SCHEMA', N'dbo',
'TABLE', N'MessageQueueErrorRecord',
'COLUMN', N'msg_body'
GO

EXEC sp_addextendedproperty
'MS_Description', N'Wrong description',
'SCHEMA', N'dbo',
'TABLE', N'MessageQueueErrorRecord',
'COLUMN', N'error_desc'
GO

EXEC sp_addextendedproperty
'MS_Description', N'Original id',
'SCHEMA', N'dbo',
'TABLE', N'MessageQueueErrorRecord',
'COLUMN', N'original_id'
GO

EXEC sp_addextendedproperty
'MS_Description', N'Push again, 0: No, 1: Yes',
'SCHEMA', N'dbo',
'TABLE', N'MessageQueueErrorRecord',
'COLUMN', N'is_re_push'
GO

EXEC sp_addextendedproperty
'MS_Description', N'Creation time',
'SCHEMA', N'dbo',
'TABLE', N'MessageQueueErrorRecord',
'COLUMN', N'created_stime'
GO

EXEC sp_addextendedproperty
'MS_Description', N'Modification time',
'SCHEMA', N'dbo',
'TABLE', N'MessageQueueErrorRecord',
'COLUMN', N'modified_stime'
GO

EXEC sp_addextendedproperty
'MS_Description', N'Delete',
'SCHEMA', N'dbo',
'TABLE', N'MessageQueueErrorRecord',
'COLUMN', N'is_del'
GO

EXEC sp_addextendedproperty
'MS_Description', N'Message queuing error logging table',
'SCHEMA', N'dbo',
'TABLE', N'MessageQueueErrorRecord'
GO


-- ----------------------------
-- Primary Key structure for table MessageQueueErrorRecord
-- ----------------------------
ALTER TABLE [dbo].[MessageQueueErrorRecord] ADD CONSTRAINT [PK_MessageQueueErrorRecord] PRIMARY KEY CLUSTERED ([id])
WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)  
ON [DFG]
GO
  • MYSQL
    

github source address

https://github.com/shawntime/shawn-basicmq

Parameter data

rabbitMQ dynamic queue implementation reference: https://blog.csdn.net/kingvin_xm/article/details/86712613

Published 2 original articles, praised 0, visited 10
Private letter follow

Tags: RabbitMQ Spring github Maven

Posted on Thu, 16 Jan 2020 05:43:16 -0500 by im8kers