Thoroughly understand etcd series: etcd cluster operation and maintenance deployment

0 album overview

etcd is an important basic component in cloud native architecture, which is hosted by CNCF incubation. etcd can be used not only as service registration and discovery, but also as key value storage Middleware in microservices and Kubernates clusters.

"Thoroughly understand etcd series of articles" will introduce etcd from its basic function practice, API interface, implementation principle, source code analysis, and implementation experience. It is expected that there will be about 20 articles, and the author will keep updating them every week. Welcome to pay attention.

1 etcd cluster deployment

In the production environment, for the high availability of the whole cluster, etcd will be deployed in the cluster normally to avoid a single point of failure. This section describes how to deploy etcd cluster. There are three mechanisms to boot etcd cluster:

  • static state
  • etcd dynamic discovery
  • DNS discovery

Statically starting an etcd cluster requires that each member knows the other member in the cluster. In many cases, the IP address of a cluster member may be unknown in advance. In these cases, you can boot the etcd cluster with the help of the discovery service.

Now we will introduce these methods separately.

2 start etcd cluster in static mode

Stand alone installation

If you want to build an etcd cluster on a machine, you can use the goreman tool.

Goreman is a multi process management tool written in Go language, which rewrites foreman, which is widely used in Ruby (foreman's original author also implemented a Go version: forego, but it is not easy to use goreman).

We need to confirm the Go installation environment, and then directly execute:

go get github.com/mattn/goreman

The compiled file is placed in $GOPATH/bin, and the $GOPATH/bin directory has been added to the system $PATH, so we can easily execute the command goreman. The following is to write the Procfile script. We start three etcd s, which correspond to the following:

HostName | ip | client interaction port | peer communication port
:-: | :-: | :-: | :-:
infra1 | 127.0.0.1 | 12379 | 12380 |
infra2 | 127.0.0.1| 22379 | 22380 |
infra3 | 127.0.0.1| 32379 | 32380 |



The Procfile script is as follows:

etcd1: etcd --name infra1 --listen-client-urls http://127.0.0.1:12379 --advertise-client-urls http://127.0.0.1:12379 --listen-peer-urls http://127.0.0.1:12380 --initial-advertise-peer-urls http://127.0.0.1:12380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state new --enable-pprof --logger=zap --log-outputs=stderr
etcd2: etcd --name infra2 --listen-client-urls http://127.0.0.1:22379 --advertise-client-urls http://127.0.0.1:22379 --listen-peer-urls http://127.0.0.1:22380 --initial-advertise-peer-urls http://127.0.0.1:22380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state new --enable-pprof --logger=zap --log-outputs=stderr
etcd3: etcd --name infra3 --listen-client-urls http://127.0.0.1:32379 --advertise-client-urls http://127.0.0.1:32379 --listen-peer-urls http://127.0.0.1:32380 --initial-advertise-peer-urls http://127.0.0.1:32380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1=http://127.0.0.1:12380,infra2=http://127.0.0.1:22380,infra3=http://127.0.0.1:32380' --initial-cluster-state new --enable-pprof --logger=zap --log-outputs=stderr


Configuration Item Description:

  • --Name: the node name in the etcd cluster, which can be freely distinguished and not repeated
  • --Listen peer URLs: the URLs used to monitor the communication between nodes. Multiple URLs can be monitored. Data interaction (such as election, data synchronization, etc.) will be carried out within the cluster through these URLs
  • --Initial advertisement peer URLs: the url recommended for communication between nodes, which will be used for communication between nodes.
  • --Listen client URLs: listen to URLs used for client communication. You can also listen to multiple URLs.
  • --Advertise client URLs: the recommended client communication url for etcd agents or etcd members to communicate with etcd nodes.
  • --Initial cluster token: etcd-cluster-1, the token value of a node. When this value is set, the cluster will generate a unique id and generate a unique id for each node. When starting a cluster with the same configuration file, as long as the token value is different, etcd clusters will not affect each other.
  • --Initial cluster: the collection of all the initial-advertisement-peer-urls in the cluster.
  • --Initial cluster state: new, the flag for creating a new cluster

Note the above script. When the etcd command is executed, it needs to be configured according to the local actual installation address. Let's start the etcd cluster.

goreman -f /opt/procfile start

Use the above command to start the etcd cluster, and check the members of the cluster after starting.

$ etcdctl --endpoints=http://localhost:22379  member list

8211f1d0f64f3269, started, infra1, http://127.0.0.1:12380, http://127.0.0.1:12379, false
91bc3c398fb3c146, started, infra2, http://127.0.0.1:22380, http://127.0.0.1:22379, false
fd422379fda50e48, started, infra3, http://127.0.0.1:32380, http://127.0.0.1:32379, false




We have successfully built a pseudo cluster on a single machine. It should be noted that when the cluster is started, we specify the members of the cluster in a static way. In the actual environment, the ip address of the members of the cluster may not be known in advance. At this time, we need to adopt the mechanism of dynamic discovery.

docker start cluster

Etcd use gcr.io/etcd-development/etcd is the main accelerator of container, quay.io/coreos/etcd As an auxiliary accelerator. Unfortunately, we can't access these two accelerators. If you can't download them, you can use the address provided by the author:

docker pull bitnami/etcd:3.4.7

Then re tag the pulled image:

docker image tag bitnami/etcd:3.4.7 quay.io/coreos/etcd:3.4.7

After the image is set, we start the etcd cluster of three nodes. The script command is as follows:

REGISTRY=quay.io/coreos/etcd

# For each machine
ETCD_VERSION=3.4.7
TOKEN=my-etcd-token
CLUSTER_STATE=new
NAME_1=etcd-node-0
NAME_2=etcd-node-1
NAME_3=etcd-node-2
HOST_1= 192.168.202.128
HOST_2= 192.168.202.129
HOST_3= 192.168.202.130
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380
DATA_DIR=/var/lib/etcd

# For node 1
THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
docker run \
  -p 2379:2379 \
  -p 2380:2380 \
  --volume=${DATA_DIR}:/etcd-data \
  --name etcd ${REGISTRY}:${ETCD_VERSION} \
  /usr/local/bin/etcd \
  --data-dir=/etcd-data --name ${THIS_NAME} \
  --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://0.0.0.0:2380 \
  --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://0.0.0.0:2379 \
  --initial-cluster ${CLUSTER} \
  --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

# For node 2
THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
docker run \
  -p 2379:2379 \
  -p 2380:2380 \
  --volume=${DATA_DIR}:/etcd-data \
  --name etcd ${REGISTRY}:${ETCD_VERSION} \
  /usr/local/bin/etcd \
  --data-dir=/etcd-data --name ${THIS_NAME} \
  --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://0.0.0.0:2380 \
  --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://0.0.0.0:2379 \
  --initial-cluster ${CLUSTER} \
  --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

# For node 3
THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
docker run \
  -p 2379:2379 \
  -p 2380:2380 \
  --volume=${DATA_DIR}:/etcd-data \
  --name etcd ${REGISTRY}:${ETCD_VERSION} \
  /usr/local/bin/etcd \
  --data-dir=/etcd-data --name ${THIS_NAME} \
  --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://0.0.0.0:2380 \
  --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://0.0.0.0:2379 \
  --initial-cluster ${CLUSTER} \
  --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}


























































Note that the above script is deployed on three machines, and each machine can execute the corresponding script. You can specify the API version at run time:

docker exec etcd /bin/sh -c "export ETCDCTL_API=3 && /usr/local/bin/etcdctl put foo bar"

The installation mode of docker is relatively simple. Readers can customize some configurations according to their needs.

3 dynamic discovery starts etcd cluster

As mentioned earlier, in a real environment, the ip of cluster members may not be known in advance. In this case, you need to use auto discovery to guide the etcd cluster instead of specifying a static configuration, which is called discovery. We start three etcds, which are as follows:

HostName | ip | client interaction port | peer communication port
:-: | :-: | :-: | :-:
etcd1 | 192.168.202.128 | 2379 | 2380 |
etcd2 | 192.168.202.129| 2379 | 2380 |
etcd3 | 192.168.202.130| 2379 | 2380 |



Principles of agreement

The Discovery service protocol helps new etcd members discover all other members during the cluster boot phase using the shared URL.

The protocol uses a new discovery token to guide a unique etcd cluster. A discovery token can only represent one etcd cluster. As long as the discovery protocol on this token is started, it cannot be used to boot another etcd cluster even if it fails halfway.

Tip: the Discovery service protocol is only used in the cluster boot phase and cannot be used for runtime reconfiguration or cluster monitoring.

Discovery protocol uses an internal etcd cluster to coordinate the bootstrap of the new cluster. First, all new members interact with the discovery service and help generate the expected list of members. After that, each new member uses this list to boot its server, which performs the same function as the -- initial cluster flag, that is, to set the member information of all clusters.

Get the token of discovery

Generate a unique token that will identify the new cluster. In the following steps, it will be used as a unique prefix in the discovery key space. A simple way is to use uuidgen:

UUID=$(uuidgen)

Specify the size of the cluster

When getting a token, you must specify the cluster size. The discovery service uses this size to know when all the members that will initially form the cluster are discovered.

curl -X PUT http://10.0.10.10:2379/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83/_config/size -d value=3

We need to put the url address http://10.0.10.10:2379/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83 as the -- discovery parameter to start etcd.

Nodes are automatically used http://10.0.10.10:2379/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83 directory for etcd registration and discovery services.

Public discovery services

When there is no etcd cluster available locally, the etcd official website provides an etcd storage address that can be accessed by the public network. We can get the directory of etcd service through the following command and use it as the -- discovery parameter.

Public discovery services discovery.etcd.io It works in the same way, but with a layer of decoration, it can extract ugly URL s, generate UUID s automatically, and provide protection for too many requests. The public discovery service still uses the etcd cluster as the data store on it.

$ curl http://discovery.etcd.io/new?size=3

http://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de


Start cluster by dynamic discovery

In etcd discovery mode, the command to start etcd is as follows:

# etcd1 start-up
$ /opt/etcd/bin/etcd  --name etcd1 --initial-advertise-peer-urls http://192.168.202.128:2380 \
  --listen-peer-urls http://192.168.202.128:2380 \
  --data-dir /opt/etcd/data \
  --listen-client-urls http://192.168.202.128:2379,http://127.0.0.1:2379 \
  --advertise-client-urls http://192.168.202.128:2379 \
  --discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de

# etcd2 start-up
 /opt/etcd/bin/etcd  --name etcd2 --initial-advertise-peer-urls http://192.168.202.129:2380 \
  --listen-peer-urls http://192.168.202.129:2380 \
  --data-dir /opt/etcd/data \
  --listen-client-urls http://192.168.202.129:2379,http://127.0.0.1:2379 \
  --advertise-client-urls http://192.168.202.129:2379 \
  --discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de

#etcd3 start
 /opt/etcd/bin/etcd  --name etcd3 --initial-advertise-peer-urls http://192.168.202.130:2380 \
    --listen-peer-urls http://192.168.202.130:2380 \
    --data-dir /opt/etcd/data \
    --listen-client-urls http://192.168.202.130:2379,http://127.0.0.1:2379 \
    --advertise-client-urls http://192.168.202.130:2379 \
    --discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de






















It should be noted that after we finish the initialization of the cluster, this information will lose its function. When you need to add nodes, you need to use etcdctl to operate. For security, each time you start a new etcd cluster, you use the new discovery token to register. In addition, if the number of nodes started during initialization exceeds the specified number, the extra nodes will be automatically converted to etcd in Proxy mode.

Result verification

After the cluster is started, verify it. Let's look at the members of the cluster:

$ /opt/etcd/bin/etcdctl member list
#The results are as follows
    40e2ac06ca1674a7, started, etcd3, http://192.168.202.130:2380, http://192.168.202.130:2379, false
    c532c5cedfe84d3c, started, etcd1, http://192.168.202.128:2380, http://192.168.202.128:2379, false
    db75d3022049742a, started, etcd2, http://192.168.202.129:2380, http://192.168.202.129:2379, false




The results are in line with the expectations, and then look at the health status of the nodes:

$ /opt/etcd/bin/etcdctl  --endpoints="http://192.168.202.128:2379,http://192.168.202.129:2379,http://192.168.202.130:2379"  endpoint  health
#The results are as follows
    http://192.168.202.128:2379 is healthy: successfully committed proposal: took = 3.157068ms
    http://192.168.202.130:2379 is healthy: successfully committed proposal: took = 3.300984ms
    http://192.168.202.129:2379 is healthy: successfully committed proposal: took = 3.263923ms





As you can see, all three nodes in the cluster are in a healthy and normal state. Starting the cluster in dynamic discovery mode succeeded.

 

4 DNS self discovery mode

etcd also supports starting using DNS SRV records. In fact, it is realized by using SRV records of DNS to continuously rotate and query. DNS SRV is a type of resource record supported in DNS database, which records the correspondence between computer and service information provided.

Dnsmasq installation

DNS services are created here using dnsmasq. Dnsmasq provides DNS caching, DHCP services and FTP services. As a domain name resolution server, dnsmasq can cache DNS requests to improve the connection speed of visited URLs. Dnsmasq is lightweight and easy to configure, suitable for personal users or networks with less than 50 hosts. It also comes with a PXE server.

When receiving a DNS request, Dnsmasq first looks for the file / etc/hosts, and then looks for / etc/resolv.conf External DNS as defined in. Configure Dnsmasq as DNS cache server, and add local intranet resolution in / etc/hosts file, so that the intranet machine will query the hosts file first, which is equivalent to sharing / etc/hosts to the whole intranet machine, so as to solve the problem of mutual recognition of the intranet machines. Instead of editing the hosts file one by one or adding a Bind DNS record, you can edit only one hosts file.

Based on the host of Centos 7 used by the author, first install Dnsmasq:

yum install dnsmasq

After installation, configure. All configurations are completed in one file / etc/dnsmasq.conf . We can also write a configuration file with any name in / etc/dnsmasq.d.

Configure upstream server address

Resolv file configures additional DNS servers upstream of Dnsmasq. If it is not enabled, the default / etc of Linux host will be used/ resolv.conf nameserver in.

$ vim /etc/dnsmasq.conf

#Add the following:
resolv-file=/etc/resolv.dnsmasq.conf
srv-host=_etcd-server._tcp.blueskykong.com,etcd1.blueskykong.com,2380,0,100
srv-host=_etcd-server._tcp.blueskykong.com,etcd2.blueskykong.com,2380,0,100
srv-host=_etcd-server._tcp.blueskykong.com,etcd3.blueskykong.com,2380,0,100






At dnsmasq.conf The corresponding domain name records in configure the three servers involved, respectively corresponding to etcd1, etcd2, etcd3.

Add forwarding DNS address in the specified file
$ vim /etc/resolv.dnsmasq.conf

nameserver 8.8.8.8
nameserver 8.8.4.4



These two free DNS services should be familiar to everyone. Readers can configure according to the local actual network.

Enable Dnsmasq resolution locally
$ vim /etc/resolv.conf

nameserver 127.0.0.1


It is easy to understand that Dnsmasq parsing is configured locally.

Add resolution record

Configure the related A record for each domain name to point to the corresponding machine IP of the etcd core node. There are three ways to add parsing records: using the system default hosts, using the custom hosts file, and using the custom conf. Here we use the simpler first method.

$ vim /etc/hosts

#Add the following content analysis
192.168.202.128 etcd1.blueskykong.com
192.168.202.129 etcd2.blueskykong.com
192.168.202.130 etcd3.blueskykong.com





Start service

service dnsmasq start

After startup, we verify:

  • The results of SRV record query on DNS server are as follows:

    <span class="katex"><annotation encoding="application x-tex"=""> dig @192.168.202.128 +noall +answer SRV _etcd-server._tcp.blueskykong.com

    _etcd-server._tcp.blueskykong.com. 0 IN SRV     100 2380 etcd2.blueskykong.com.
    _etcd-server._tcp.blueskykong.com. 0 IN SRV     100 2380 etcd1.blueskykong.com.
    _etcd-server._tcp.blueskykong.com. 0 IN SRV     100 2380 etcd3.blueskykong.com.
    </code></pre></li>
    <li><p>Make query domain name resolution result</p>
    <pre><code><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8888799999999999em;vertical-align:-0.19444em;"><span class="mord mathit">d<span class="mord mathit">i<span class="mord mathit" style="margin-right:0.03588em;">g<span class="mord">@<span class="mord">1<span class="mord">9<span class="mord">2<span class="mord">.<span class="mord">1<span class="mord">6<span class="mord">8<span class="mord">.<span class="mord">2<span class="mord">0<span class="mord">2<span class="mord">.<span class="mord">1<span class="mord">2<span class="mord">8<span class="mspace" style="margin-right:0.2222222222222222em;"><span class="mbin">+<span class="mspace" style="margin-right:0.2222222222222222em;"><span class="base"><span class="strut" style="height:0.77777em;vertical-align:-0.08333em;"><span class="mord mathit">n<span class="mord mathit">o<span class="mord mathit">a<span class="mord mathit" style="margin-right:0.01968em;">l<span class="mord mathit" style="margin-right:0.01968em;">l<span class="mspace" style="margin-right:0.2222222222222222em;"><span class="mbin">+<span class="mspace" style="margin-right:0.2222222222222222em;"><span class="base"><span class="strut" style="height:0.8888799999999999em;vertical-align:-0.19444em;"><span class="mord mathit">a<span class="mord mathit">n<span class="mord mathit">s<span class="mord mathit" style="margin-right:0.02691em;">w<span class="mord mathit">e<span class="mord mathit" style="margin-right:0.02778em;">r<span class="mord mathit" style="margin-right:0.05764em;">S<span class="mord mathit" style="margin-right:0.00773em;">R<span class="mord mathit" style="margin-right:0.22222em;">V<span class="mord">¨<span class="mord mathit" style="margin-right:0.05764em;">E<span class="mord">9<span class="mord">5<span class="mord mathit" style="margin-right:0.05764em;">E<span class="mord mathit">e<span class="mord mathit">t<span class="mord mathit">c<span class="mord mathit">d<span class="mord">¨<span class="mord mathit" style="margin-right:0.05764em;">E<span class="mord">4<span class="mord">5<span class="mord mathit" style="margin-right:0.05764em;">E<span class="mord mathit">s<span class="mord mathit">e<span class="mord mathit" style="margin-right:0.02778em;">r<span class="mord mathit" style="margin-right:0.03588em;">v<span class="mord mathit">e<span class="mord mathit" style="margin-right:0.02778em;">r<span class="mord">.<span class="mord">¨<span class="mord mathit" style="margin-right:0.05764em;">E<span class="mord">9<span class="mord">5<span class="mord mathit" style="margin-right:0.05764em;">E<span class="mord mathit">t<span class="mord mathit">c<span class="mord mathit">p<span class="mord">.<span class="mord mathit">b<span class="mord mathit" style="margin-right:0.01968em;">l<span class="mord mathit">u<span class="mord mathit">e<span class="mord mathit">s<span class="mord mathit" style="margin-right:0.03148em;">k<span class="mord mathit" style="margin-right:0.03588em;">y<span class="mord mathit" style="margin-right:0.03148em;">k<span class="mord mathit">o<span class="mord mathit">n<span class="mord mathit" style="margin-right:0.03588em;">g<span class="mord">.<span class="mord mathit">c<span class="mord mathit">o<span class="mord mathit">m<span class="mord">¨<span class="mord mathit" style="margin-right:0.05764em;">E<span class="mord">9<span class="mord">5<span class="mord mathit" style="margin-right:0.05764em;">E<span class="mord mathit">e<span class="mord mathit">t<span class="mord mathit">c<span class="mord mathit">d<span class="mord">¨<span class="mord mathit" style="margin-right:0.05764em;">E<span class="mord">4<span class="mord">5<span class="mord mathit" style="margin-right:0.05764em;">E<span class="mord mathit">s<span class="mord mathit">e<span class="mord mathit" style="margin-right:0.02778em;">r<span class="mord mathit" style="margin-right:0.03588em;">v<span class="mord mathit">e<span class="mord mathit" style="margin-right:0.02778em;">r<span class="mord">.<span class="mord">¨<span class="mord mathit" style="margin-right:0.05764em;">E<span class="mord">9<span class="mord">5<span class="mord mathit" style="margin-right:0.05764em;">E<span class="mord mathit">t<span class="mord mathit">c<span class="mord mathit">p<span class="mord">.<span class="mord mathit">b<span class="mord mathit" style="margin-right:0.01968em;">l<span class="mord mathit">u<span class="mord mathit">e<span class="mord mathit">s<span class="mord mathit" style="margin-right:0.03148em;">k<span class="mord mathit" style="margin-right:0.03588em;">y<span class="mord mathit" style="margin-right:0.03148em;">k<span class="mord mathit">o<span class="mord mathit">n<span class="mord mathit" style="margin-right:0.03588em;">g<span class="mord">.<span class="mord mathit">c<span class="mord mathit">o<span class="mord mathit">m<span class="mord">.<span class="mord">0<span class="mord mathit" style="margin-right:0.07847em;">I<span class="mord mathit" style="margin-right:0.10903em;">N<span class="mord mathit" style="margin-right:0.05764em;">S<span class="mord mathit" style="margin-right:0.00773em;">R<span class="mord mathit" style="margin-right:0.22222em;">V<span class="mord">0<span class="mord">1<span class="mord">0<span class="mord">0<span class="mord">2<span class="mord">3<span class="mord">8<span class="mord">0<span class="mord mathit">e<span class="mord mathit">t<span class="mord mathit">c<span class="mord mathit">d<span class="mord">2<span class="mord">.<span class="mord mathit">b<span class="mord mathit" style="margin-right:0.01968em;">l<span class="mord mathit">u<span class="mord mathit">e<span class="mord mathit">s<span class="mord mathit" style="margin-right:0.03148em;">k<span class="mord mathit" style="margin-right:0.03588em;">y<span class="mord mathit" style="margin-right:0.03148em;">k<span class="mord mathit">o<span class="mord mathit">n<span class="mord mathit" style="margin-right:0.03588em;">g<span class="mord">.<span class="mord mathit">c<span class="mord mathit">o<span class="mord mathit">m<span class="mord">.<span class="mord">¨<span class="mord mathit" style="margin-right:0.05764em;">E<span class="mord">9<span class="mord">5<span class="mord mathit" style="margin-right:0.05764em;">E<span class="mord mathit">e<span class="mord mathit">t<span class="mord mathit">c<span class="mord mathit">d<span class="mord">¨<span class="mord mathit" style="margin-right:0.05764em;">E<span class="mord">4<span class="mord">5<span class="mord mathit" style="margin-right:0.05764em;">E<span class="mord mathit">s<span class="mord mathit">e<span class="mord mathit" style="margin-right:0.02778em;">r<span class="mord mathit" style="margin-right:0.03588em;">v<span class="mord mathit">e<span class="mord mathit" style="margin-right:0.02778em;">r<span class="mord">.<span class="mord">¨<span class="mord mathit" style="margin-right:0.05764em;">E<span class="mord">9<span class="mord">5<span class="mord mathit" style="margin-right:0.05764em;">E<span class="mord mathit">t<span class="mord mathit">c<span class="mord mathit">p<span class="mord">.<span class="mord mathit">b<span class="mord mathit" style="margin-right:0.01968em;">l<span class="mord mathit">u<span class="mord mathit">e<span class="mord mathit">s<span class="mord mathit" style="margin-right:0.03148em;">k<span class="mord mathit" style="margin-right:0.03588em;">y<span class="mord mathit" style="margin-right:0.03148em;">k<span class="mord mathit">o<span class="mord mathit">n<span class="mord mathit" style="margin-right:0.03588em;">g<span class="mord">.<span class="mord mathit">c<span class="mord mathit">o<span class="mord mathit">m<span class="mord">.<span class="mord">0<span class="mord mathit" style="margin-right:0.07847em;">I<span class="mord mathit" style="margin-right:0.10903em;">N<span class="mord mathit" style="margin-right:0.05764em;">S<span class="mord mathit" style="margin-right:0.00773em;">R<span class="mord mathit" style="margin-right:0.22222em;">V<span class="mord">0<span class="mord">1<span class="mord">0<span class="mord">0<span class="mord">2<span class="mord">3<span class="mord">8<span class="mord">0<span class="mord mathit">e<span class="mord mathit">t<span class="mord mathit">c<span class="mord mathit">d<span class="mord">1<span class="mord">.<span class="mord mathit">b<span class="mord mathit" style="margin-right:0.01968em;">l<span class="mord mathit">u<span class="mord mathit">e<span class="mord mathit">s<span class="mord mathit" style="margin-right:0.03148em;">k<span class="mord mathit" style="margin-right:0.03588em;">y<span class="mord mathit" style="margin-right:0.03148em;">k<span class="mord mathit">o<span class="mord mathit">n<span class="mord mathit" style="margin-right:0.03588em;">g<span class="mord">.<span class="mord mathit">c<span class="mord mathit">o<span class="mord mathit">m<span class="mord">.<span class="mord">¨<span class="mord mathit" style="margin-right:0.05764em;">E<span class="mord">9<span class="mord">5<span class="mord mathit" style="margin-right:0.05764em;">E<span class="mord mathit">e<span class="mord mathit">t<span class="mord mathit">c<span class="mord mathit">d<span class="mord">¨<span class="mord mathit" style="margin-right:0.05764em;">E<span class="mord">4<span class="mord">5<span class="mord mathit" style="margin-right:0.05764em;">E<span class="mord mathit">s<span class="mord mathit">e<span class="mord mathit" style="margin-right:0.02778em;">r<span class="mord mathit" style="margin-right:0.03588em;">v<span class="mord mathit">e<span class="mord mathit" style="margin-right:0.02778em;">r<span class="mord">.<span class="mord">¨<span class="mord mathit" style="margin-right:0.05764em;">E<span class="mord">9<span class="mord">5<span class="mord mathit" style="margin-right:0.05764em;">E<span class="mord mathit">t<span class="mord mathit">c<span class="mord mathit">p<span class="mord">.<span class="mord mathit">b<span class="mord mathit" style="margin-right:0.01968em;">l<span class="mord mathit">u<span class="mord mathit">e<span class="mord mathit">s<span class="mord mathit" style="margin-right:0.03148em;">k<span class="mord mathit" style="margin-right:0.03588em;">y<span class="mord mathit" style="margin-right:0.03148em;">k<span class="mord mathit">o<span class="mord mathit">n<span class="mord mathit" style="margin-right:0.03588em;">g<span class="mord">.<span class="mord mathit">c<span class="mord mathit">o<span class="mord mathit">m<span class="mord">.<span class="mord">0<span class="mord mathit" style="margin-right:0.07847em;">I<span class="mord mathit" style="margin-right:0.10903em;">N<span class="mord mathit" style="margin-right:0.05764em;">S<span class="mord mathit" style="margin-right:0.00773em;">R<span class="mord mathit" style="margin-right:0.22222em;">V<span class="mord">0<span class="mord">1<span class="mord">0<span class="mord">0<span class="mord">2<span class="mord">3<span class="mord">8<span class="mord">0<span class="mord mathit">e<span class="mord mathit">t<span class="mord mathit">c<span class="mord mathit">d<span class="mord">3<span class="mord">.<span class="mord mathit">b<span class="mord mathit" style="margin-right:0.01968em;">l<span class="mord mathit">u<span class="mord mathit">e<span class="mord mathit">s<span class="mord mathit" style="margin-right:0.03148em;">k<span class="mord mathit" style="margin-right:0.03588em;">y<span class="mord mathit" style="margin-right:0.03148em;">k<span class="mord mathit">o<span class="mord mathit">n<span class="mord mathit" style="margin-right:0.03588em;">g<span class="mord">.<span class="mord mathit">c<span class="mord mathit">o<span class="mord mathit">m<span class="mord">.<span class="mspace" style="margin-right:0.2777777777777778em;"><span class="mrel"><<span class="mspace" style="margin-right:0.2777777777777778em;"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"><span class="mord">/<span class="mord mathit">c<span class="mord mathit">o<span class="mord mathit">d<span class="mord mathit">e<span class="mspace" style="margin-right:0.2777777777777778em;"><span class="mrel">><span class="base"><span class="strut" style="height:0.5782em;vertical-align:-0.0391em;"><span class="mrel"><<span class="mspace" style="margin-right:0.2777777777777778em;"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"><span class="mord">/<span class="mord mathit">p<span class="mord mathit" style="margin-right:0.02778em;">r<span class="mord mathit">e<span class="mspace" style="margin-right:0.2777777777777778em;"><span class="mrel">><span class="base"><span class="strut" style="height:0.5782em;vertical-align:-0.0391em;"><span class="mrel"><<span class="mspace" style="margin-right:0.2777777777777778em;"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"><span class="mord">/<span class="mord mathit" style="margin-right:0.01968em;">l<span class="mord mathit">i<span class="mspace" style="margin-right:0.2777777777777778em;"><span class="mrel">><span class="base"><span class="strut" style="height:0.5782em;vertical-align:-0.0391em;"><span class="mrel"><<span class="mspace" style="margin-right:0.2777777777777778em;"><span class="base"><span class="strut" style="height:0.73354em;vertical-align:-0.0391em;"><span class="mord mathit" style="margin-right:0.01968em;">l<span class="mord mathit">i<span class="mspace" style="margin-right:0.2777777777777778em;"><span class="mrel">><span class="base"><span class="strut" style="height:0.5782em;vertical-align:-0.0391em;"><span class="mrel"><<span class="mspace" style="margin-right:0.2777777777777778em;"><span class="base"><span class="strut" style="height:0.7335400000000001em;vertical-align:-0.19444em;"><span class="mord mathit">p<span class="mspace" style="margin-right:0.2777777777777778em;"><span class="mrel">><span class="mspace" style="margin-right:0.2777777777777778em;"><span class="base"><span class="strut" style="height:0.5782em;vertical-align:-0.0391em;"><span class="mord cjk_fallback">send<span class="mord cjk_fallback">check<span class="mord cjk_fallback">Inquiry<span class="mord cjk_fallback">field<span class="mord cjk_fallback">name<span class="mord cjk_fallback">solution<span class="mord cjk_fallback">Analysis<span class="mord cjk_fallback">junction<span class="mord cjk_fallback">fruit<span class="mspace" style="margin-right:0.2777777777777778em;"><span class="mrel"><<span class="mspace" style="margin-right:0.2777777777777778em;"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"><span class="mord">/<span class="mord mathit">p<span class="mspace" style="margin-right:0.2777777777777778em;"><span class="mrel">><span class="base"><span class="strut" style="height:0.5782em;vertical-align:-0.0391em;"><span class="mrel"><<span class="mspace" style="margin-right:0.2777777777777778em;"><span class="base"><span class="strut" style="height:0.7335400000000001em;vertical-align:-0.19444em;"><span class="mord mathit">p<span class="mord mathit" style="margin-right:0.02778em;">r<span class="mord mathit">e<span class="mspace" style="margin-right:0.2777777777777778em;"><span class="mrel">><span class="base"><span class="strut" style="height:0.5782em;vertical-align:-0.0391em;"><span class="mrel"><<span class="mspace" style="margin-right:0.2777777777777778em;"><span class="base"><span class="strut" style="height:0.73354em;vertical-align:-0.0391em;"><span class="mord mathit">c<span class="mord mathit">o<span class="mord mathit">d<span class="mord mathit">e<span class="mspace" style="margin-right:0.2777777777777778em;"><span class="mrel">> dig @192.168.202.128 +noall +answer etcd1.blueskykong.com etcd2.blueskykong.com etcd3.blueskykong.com

    etcd1.blueskykong.com.  0       IN      A       192.168.202.128
    etcd2.blueskykong.com.  0       IN      A       192.168.202.129
    etcd3.blueskykong.com.  0       IN      A       192.168.202.130











At this point, we have successfully installed Dnsmasq. Next, we start the etcd cluster based on DNS discovery.

Start cluster

After the above two steps of DNS configuration, you can use DNS to start the etcd cluster. Etcd needs to be removed_ INITIAL_ Cluster configuration (for static service discovery) and specify DNS SRV domain name (ETCD_DISCOVERY_SRV). Configure the url parameter of DNS resolution as - Discovery SRV, where the start command of etcd1 node is as follows:

$ /opt/etcd/bin/etcd   --name etcd1 \
--discovery-srv blueskykong.com \
--initial-advertise-peer-urls http://etcd1.blueskykong.com:2380 \
--initial-cluster-token etcd-cluster-1 \
--data-dir /opt/etcd/data \
--initial-cluster-state new \
--advertise-client-urls http://etcd1.blueskykong.com:2379 \
--listen-client-urls http://0.0.0.0:2379 \
--listen-peer-urls http://0.0.0.0:2380








Etcd cluster members can broadcast using domain names or IP addresses, and the process of starting will resolve DNS records. --The resolution address in initial advertisement peer URLs must match the resolution address in the SRV target. The etcd member reads the resolved address to see if it belongs to the cluster defined in the SRV record.

We verify the correctness of starting the cluster based on DNS discovery and view the list of cluster members:

$ /opt/etcd/bin/etcdctl member list

#The results are as follows:
40e2ac06ca1674a7, started, etcd3, http://192.168.202.130:2380, http://etcd3.blueskykong.com:2379, false
c532c5cedfe84d3c, started, etcd1, http://192.168.202.128:2380, http://etcd1.blueskykong.com:2379, false
db75d3022049742a, started, etcd2, http://192.168.202.129:2380, http://etcd2.blueskykong.com:2379, false





As you can see, the result output etcd cluster has three members, which is in line with the expectation. Next, we use the IP address to continue to verify the status of the cluster nodes.

$ /opt/etcd/bin/etcdctl  --endpoints="http://192.168.202.128:2379,http://192.168.202.129:2379,http://192.168.202.130:2379"  endpoint  health

#The results are as follows:
http://192.168.202.129:2379 is healthy: successfully committed proposal: took = 2.933555ms
http://192.168.202.128:2379 is healthy: successfully committed proposal: took = 7.252799ms
http://192.168.202.130:2379 is healthy: successfully committed proposal: took = 7.415843ms





More etcd cluster operations, readers can try their own, the author does not launch here one by one.

5 Summary

Based on the previous article, this paper mainly introduces the start-up methods of etcd cluster: static monomer, static docker, dynamic discovery and DNS discovery. So many installation postures are for our actual use. In the next article, we will go into the use of etcdctl.

Subscribe to the latest article. Welcome to my official account.

Recommended reading

  1. Comparison of etcd with Zookeeper, Consul and other k-v components
  2. A thorough understanding of etcd series (1): initial understanding of etcd
  3. A thorough understanding of etcd series (2): various installation postures of etcd

reference resources

etcd docs

Tags: DNS Docker vim curl

Posted on Sun, 07 Jun 2020 04:54:53 -0400 by Kyrst