Big data Kafka

1.1 function of message queue

1) Decoupling
It allows you to extend or modify the processes on both sides independently, as long as you ensure that they comply with the same interface constraints.
2) Recoverability
Failure of some components of the system will not affect the whole system. Message queuing reduces the coupling between processes
Even if a message processing process hangs, the messages added to the queue can still be processed after the system recovers.
3) Buffering helps to control and optimize the speed of data flow through the system and solve the inconsistency between the processing speed of production messages and consumption messages.
4) Flexibility & peak processing power
In the case of a sharp increase in traffic, applications still need to continue to play a role, but such burst traffic is not common. It would be a huge waste to put resources on standby to handle such peak visits. Using message queuing
It enables key components to withstand sudden access pressure without completely crashing due to sudden overload requests.
5) Asynchronous communication
Many times, users do not want or need to process messages immediately. Message queuing provides an asynchronous processing mechanism that allows users to put a message on the queue without processing it immediately. Put as many messages into the queue as you want, and then process them when needed.

1.2 two modes of message queue

(1) Point to point mode (one-to-one, consumers actively pull data, and the message is cleared after receiving the message)
The message producer sends the message to the queue, and then the message consumer takes it out of the queue and consumes the message. After the message is consumed, there is no storage in the queue, so it is impossible for the message consumer to consume the consumed message.
Queue supports multiple consumers, but for a message, only one consumer can consume.

(2) Publish / subscribe mode (one to many, messages will not be cleared after consumer consumption data)
The message producer (publisher) publishes the message to the topic, and multiple message consumers (subscribers) consume the message at the same time
Rest.
There are two modes for publishing and subscribing:
1. Consumers actively pull data
2. Queues actively push data -- consumers' consumption rates are not equal, which may lead to waste of service resources and service collapse.

2.1 Kafka architecture

1. Producer consumption news
2. Kafks cluster management message
   1. kafka cluster is a process running on the server.
   2. The Kafka cluster partitions messages according to different topics.
   3. Kafka will make redundant backup of data (not on the same server), leader and follower (data backup)
3. Consumer News
   1. Consumer group: a partition can only be consumed by consumers in the same consumer group.
   Consumer group, improve consumption ability.
4. Zookeeper registration message
   1. Kafka registry
   2. Consumer consumption information is stored in zk, which is convenient for consumers to read consumption information from zk after hanging up.
   Before version 0.9, it exists in zk, and after version 0.9, it is saved in a topic in Kafka. Save to disk (7 days by default)

2.1 docker deploying Kafka cluster

2.1.1 docker compose installation

1,sudo curl -L "https://github.com/docker/compose/releases/download/1.28.2/docker-compose- ( u n a m e − s ) − (uname -s)- (uname−s)−(uname -m)" -o /usr/local/bin/docker-compose
2,chmod +x /usr/local/bin/docker-compose

2.1.2 Kafka cluster installation

1. Create virtual network
docker network create --driver bridge --subnet 172.23.0.0/16 --gateway 172.23.0.1 zoo_kafka
docker-compose.yml is as follows:

version: '2' services: zoo1: image: zookeeper:3.4 # Image name restart: always # Automatic restart in case of error hostname: zoo1 container_name: zoo1 privileged: true ports: # port - 2181:2181 volumes: # Mount data volume - ./zoo1/data:/data - ./zoo1/datalog:/datalog environment: TZ: Asia/Shanghai ZOO_MY_ID: 1 # Node ID ZOO_PORT: 2181 # zookeeper port number ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888 # zookeeper node list networks: default: ipv4_address: 172.23.0.11 zoo2: image: zookeeper:3.4 restart: always hostname: zoo2 container_name: zoo2 privileged: true ports: - 2182:2181 volumes: - ./zoo2/data:/data - ./zoo2/datalog:/datalog environment: TZ: Asia/Shanghai ZOO_MY_ID: 2 ZOO_PORT: 2181 ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888 networks: default: ipv4_address: 172.23.0.12 zoo3: image: zookeeper:3.4 restart: always hostname: zoo3 container_name: zoo3 privileged: true ports: - 2183:2181 volumes: - ./zoo3/data:/data - ./zoo3/datalog:/datalog environment: TZ: Asia/Shanghai ZOO_MY_ID: 3 ZOO_PORT: 2181 ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888 networks: default: ipv4_address: 172.23.0.13 networks: default: external: name: zoo_kafka

Kafka: docker-compose.yml content:
Docker compose up - d start

version: '2' services: broker1: image: wurstmeister/kafka restart: always hostname: broker1 container_name: broker1 privileged: true ports: - "9091:9092" environment: KAFKA_BROKER_ID: 1 KAFKA_LISTENERS: PLAINTEXT://broker1:9092 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker1:9092 KAFKA_ADVERTISED_HOST_NAME: broker1 KAFKA_ADVERTISED_PORT: 9092 KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181 #JMX_PORT: 9987 volumes: - /var/run/docker.sock:/var/run/docker.sock - ./broker1:/kafka/kafka\-logs\-broker1 external_links: - zoo1 - zoo2 - zoo3 networks: default: ipv4_address: 172.23.0.14 broker2: image: wurstmeister/kafka restart: always hostname: broker2 container_name: broker2 privileged: true ports: - "9092:9092" environment: KAFKA_BROKER_ID: 2 KAFKA_LISTENERS: PLAINTEXT://broker2:9092 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker2:9092 KAFKA_ADVERTISED_HOST_NAME: broker2 KAFKA_ADVERTISED_PORT: 9092 KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181 #JMX_PORT: 9988 volumes: - /var/run/docker.sock:/var/run/docker.sock - ./broker2:/kafka/kafka\-logs\-broker2 external_links: # Connect container s other than this compose file - zoo1 - zoo2 - zoo3 networks: default: ipv4_address: 172.23.0.15 broker3: image: wurstmeister/kafka restart: always hostname: broker3 container_name: broker3 privileged: true ports: - "9093:9092" environment: KAFKA_BROKER_ID: 3 KAFKA_LISTENERS: PLAINTEXT://broker3:9092 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker3:9092 KAFKA_ADVERTISED_HOST_NAME: broker3 KAFKA_ADVERTISED_PORT: 9092 KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181 #JMX_PORT: 9989 volumes: - /var/run/docker.sock:/var/run/docker.sock - ./broker3:/kafka/kafka\-logs\-broker3 external_links: # Connect container s other than this compose file - zoo1 - zoo2 - zoo3 networks: default: ipv4_address: 172.23.0.16 kafka-manager: image: sheepkiller/kafka-manager:latest restart: always container_name: kafka-manager hostname: kafka-manager ports: - "9000:9000" links: # Connect the container created by this compose file - broker1 - broker2 - broker3 external_links: # Connect container s other than this compose file - zoo1 - zoo2 - zoo3 environment: ZK_HOSTS: zoo1:2181,zoo2:2181,zoo3:2181 KAFKA_BROKERS: broker1:9092,broker2:9092,broker3:9092 APPLICATION_SECRET: letmein KM_ARGS: -Djava.net.preferIPv4Stack=true networks: default: ipv4_address: 172.23.0.10 networks: default: external: # Use created network name: zoo_kafka

3.1 kafka command line test

Theme functions help us partition

1. kafka-topics.sh --list --zookeeper zoo1:2181 -- View topics
2. Kafka-topics.sh -- zookeeper zoo1:2181 -- create -- replication factor 3 -- partitions 1 - topic first -- create a topic
   Option Description:
   – topic defines the topic name
   – the number of replicas defined by replication factor cannot exceed the number of machines, otherwise an error will be reported.
   – partitions defines the number of partitions
3. kafka-topics.sh --describe --topic frist --zookeeper zoo1:2181 -- describe a topic
   
4. kafka-topics.sh --zookeeper zook1:2181 --delete --topic first -- delete topic

Add new comment

0 comments