Missile intercept, precision defense.
background
Interceptor: a set of event based pluggable logic processing chains without modifying the application business logic;
An interceptor similar to spring MVC:
These are through the configuration of interceptors, inserted into the application, to achieve pluggable business logic modification;
kafka started to introduce interceptors in version 0.10.0.0. It can be divided into producer blocker and consumer blocker. In the way of responsibility chain, multiple blockers are arranged as one big blocker.
Configuration method: configuration parameters
Properties props = new Properties(); List<String> interceptors = new ArrayList<>(); interceptors.add("com.yourcompany.kafkaproject.interceptors.AddTimestampInterceptor"); // Interceptor 1 interceptors.add("com.yourcompany.kafkaproject.interceptors.UpdateCounterInterceptor"); // Interceptor 2 props.put(ProducerConfig.INTERCEPTOR_CLASSES_CONFIG, interceptors); ......
Note: to configure an interceptor, you need to make a fully qualified name of the interceptor, and ensure that the producer or consumer client can correctly load it into the configured interceptor;
Through the implementation of interceptor, all producers and consumers are forced to configure the interceptor to realize the function of message audit|
Producer interceptor
The interceptor needs to implement org.apache.kafka.clients.producer.producer interceptor
Consumer blocker
org.apache.kafka.clients.consumer.ConsumerInterceptor
Practical operation
Achieve end-to-end performance monitoring:
Process:
Producer Code:
public class AvgLatencyProducerInterceptor implements ProducerInterceptor<String, String> { private Jedis jedis; // Omit Jedis initialization @Override public ProducerRecord<String, String> onSend(ProducerRecord<String, String> record) { jedis.incr("totalSentMessage"); return record; } @Override public void onAcknowledgement(RecordMetadata metadata, Exception exception) { } @Override public void close() { } @Override public void configure(Map<java.lang.String, ?> configs) { }
Consumer code:
public class AvgLatencyConsumerInterceptor implements ConsumerInterceptor<String, String> { private Jedis jedis; //Omit Jedis initialization @Override public ConsumerRecords<String, String> onConsume(ConsumerRecords<String, String> records) { long lantency = 0L; for (ConsumerRecord<String, String> record : records) { lantency += (System.currentTimeMillis() - record.timestamp()); } jedis.incrBy("totalLatency", lantency); long totalLatency = Long.parseLong(jedis.get("totalLatency")); long totalSentMsgs = Long.parseLong(jedis.get("totalSentMessage")); jedis.set("avgLatency", String.valueOf(totalLatency / totalSentMsgs)); return records; } @Override public void onCommit(Map<TopicPartition, OffsetAndMetadata> offsets) { } @Override public void close() { } @Override public void configure(Map<String, ?> configs)
Configuration to the interceptor to the corresponding producer and consumer objects, that is to simply implement the end-to-end performance statistics of the average message delay.
Summary
Analogy AOP is the core function provided by Spring, that is, aspect oriented programming, which can put security, audit and performance related functions irrelevant to business logic into aspect enhancement.
Some pluggable enhancements to Kafka can be implemented through interceptors.
This article introduces the usage of kafka's interceptor, and shows the specific usage through examples. I hope that this point can be used to expand the kafka used by the team when doing some enhancements.
Original is not easy, attention is valuable, forwarding price is higher! Reprint please indicate the source, let us communicate with each other, make progress together, welcome to communicate.
I will continue to share Java software programming knowledge and programmers' occupation career development. Welcome to my attention. I have compiled all kinds of resources for programming and learning these years, and pay attention to the official account "Li Fuchun continuous output", send "learning materials" to share with you!