How to realize the pluggable enhancement of kafka?

Missile intercept, precision defense.

background

Interceptor: a set of event based pluggable logic processing chains without modifying the application business logic;
An interceptor similar to spring MVC:

These are through the configuration of interceptors, inserted into the application, to achieve pluggable business logic modification;

kafka started to introduce interceptors in version 0.10.0.0. It can be divided into producer blocker and consumer blocker. In the way of responsibility chain, multiple blockers are arranged as one big blocker.

Configuration method: configuration parameters

Properties props = new Properties();
List<String> interceptors = new ArrayList<>();
interceptors.add("com.yourcompany.kafkaproject.interceptors.AddTimestampInterceptor"); // Interceptor 1
interceptors.add("com.yourcompany.kafkaproject.interceptors.UpdateCounterInterceptor"); // Interceptor 2
props.put(ProducerConfig.INTERCEPTOR_CLASSES_CONFIG, interceptors);
......

Note: to configure an interceptor, you need to make a fully qualified name of the interceptor, and ensure that the producer or consumer client can correctly load it into the configured interceptor;

Through the implementation of interceptor, all producers and consumers are forced to configure the interceptor to realize the function of message audit|

Producer interceptor

The interceptor needs to implement org.apache.kafka.clients.producer.producer interceptor

Consumer blocker

org.apache.kafka.clients.consumer.ConsumerInterceptor

Practical operation

Achieve end-to-end performance monitoring:

Process:

Producer Code:

public class AvgLatencyProducerInterceptor implements ProducerInterceptor<String, String> {


    private Jedis jedis; // Omit Jedis initialization


    @Override
    public ProducerRecord<String, String> onSend(ProducerRecord<String, String> record) {
        jedis.incr("totalSentMessage");
        return record;
    }


    @Override
    public void onAcknowledgement(RecordMetadata metadata, Exception exception) {
    }


    @Override
    public void close() {
    }


    @Override
    public void configure(Map<java.lang.String, ?> configs) {
    }

Consumer code:

public class AvgLatencyConsumerInterceptor implements ConsumerInterceptor<String, String> {


    private Jedis jedis; //Omit Jedis initialization


    @Override
    public ConsumerRecords<String, String> onConsume(ConsumerRecords<String, String> records) {
        long lantency = 0L;
        for (ConsumerRecord<String, String> record : records) {
            lantency += (System.currentTimeMillis() - record.timestamp());
        }
        jedis.incrBy("totalLatency", lantency);
        long totalLatency = Long.parseLong(jedis.get("totalLatency"));
        long totalSentMsgs = Long.parseLong(jedis.get("totalSentMessage"));
        jedis.set("avgLatency", String.valueOf(totalLatency / totalSentMsgs));
        return records;
    }


    @Override
    public void onCommit(Map<TopicPartition, OffsetAndMetadata> offsets) {
    }


    @Override
    public void close() {
    }


    @Override
    public void configure(Map<String, ?> configs) 

Configuration to the interceptor to the corresponding producer and consumer objects, that is to simply implement the end-to-end performance statistics of the average message delay.

Summary

Analogy AOP is the core function provided by Spring, that is, aspect oriented programming, which can put security, audit and performance related functions irrelevant to business logic into aspect enhancement.
Some pluggable enhancements to Kafka can be implemented through interceptors.

This article introduces the usage of kafka's interceptor, and shows the specific usage through examples. I hope that this point can be used to expand the kafka used by the team when doing some enhancements.

Original is not easy, attention is valuable, forwarding price is higher! Reprint please indicate the source, let us communicate with each other, make progress together, welcome to communicate.
I will continue to share Java software programming knowledge and programmers' occupation career development. Welcome to my attention. I have compiled all kinds of resources for programming and learning these years, and pay attention to the official account "Li Fuchun continuous output", send "learning materials" to share with you!

Tags: Java Jedis kafka Programming Spring

Posted on Wed, 06 May 2020 04:02:49 -0400 by mort