Missile intercept, precision defense.

background

Interceptor: a set of event based pluggable logic processing chains without modifying the application business logic;
An interceptor similar to spring MVC:

These are through the configuration of interceptors, inserted into the application, to achieve pluggable business logic modification;

kafka started to introduce interceptors in version 0.10.0.0. It can be divided into producer blocker and consumer blocker. In the way of responsibility chain, multiple blockers are arranged as one big blocker.

Configuration method: configuration parameters

Properties props = new Properties();
List<String> interceptors = new ArrayList<>();
interceptors.add("com.yourcompany.kafkaproject.interceptors.AddTimestampInterceptor"); // Interceptor 1
interceptors.add("com.yourcompany.kafkaproject.interceptors.UpdateCounterInterceptor"); // Interceptor 2
props.put(ProducerConfig.INTERCEPTOR_CLASSES_CONFIG, interceptors);
......

Note: to configure an interceptor, you need to make a fully qualified name of the interceptor, and ensure that the producer or consumer client can correctly load it into the configured interceptor;

Through the implementation of interceptor, all producers and consumers are forced to configure the interceptor to realize the function of message audit|

Producer interceptor

The interceptor needs to implement org.apache.kafka.clients.producer.producer interceptor

Consumer blocker

org.apache.kafka.clients.consumer.ConsumerInterceptor

Practical operation

Achieve end-to-end performance monitoring:

Process:

Producer Code:

public class AvgLatencyProducerInterceptor implements ProducerInterceptor<String, String> {


    private Jedis jedis; // Omit Jedis initialization


    @Override
    public ProducerRecord<String, String> onSend(ProducerRecord<String, String> record) {
        jedis.incr("totalSentMessage");
        return record;
    }


    @Override
    public void onAcknowledgement(RecordMetadata metadata, Exception exception) {
    }


    @Override
    public void close() {
    }


    @Override
    public void configure(Map<java.lang.String, ?> configs) {
    }

Consumer code:

public class AvgLatencyConsumerInterceptor implements ConsumerInterceptor<String, String> {


    private Jedis jedis; //Omit Jedis initialization


    @Override
    public ConsumerRecords<String, String> onConsume(ConsumerRecords<String, String> records) {
        long lantency = 0L;
        for (ConsumerRecord<String, String> record : records) {
            lantency += (System.currentTimeMillis() - record.timestamp());
        }
        jedis.incrBy("totalLatency", lantency);
        long totalLatency = Long.parseLong(jedis.get("totalLatency"));
        long totalSentMsgs = Long.parseLong(jedis.get("totalSentMessage"));
        jedis.set("avgLatency", String.valueOf(totalLatency / totalSentMsgs));
        return records;
    }


    @Override
    public void onCommit(Map<TopicPartition, OffsetAndMetadata> offsets) {
    }


    @Override
    public void close() {
    }


    @Override
    public void configure(Map<String, ?> configs) 

Configuration to the interceptor to the corresponding producer and consumer objects, that is to simply implement the end-to-end performance statistics of the average message delay.

Summary

Analogy AOP is the core function provided by Spring, that is, aspect oriented programming, which can put security, audit and performance related functions irrelevant to business logic into aspect enhancement.
Some pluggable enhancements to Kafka can be implemented through interceptors.

This article introduces the usage of kafka's interceptor, and shows the specific usage through examples. I hope that this point can be used to expand the kafka used by the team when doing some enhancements.