1, Introduction to single master cluster deployment

Installation package used to build k8s cluster: (my version of installation package)

Set up a node server: (three nodes)

Master: 192.168.66.130/24

Software to be installed: Kube apiserver Kube Controller Manager Kube scheduler etcd

Node01: 192.168.66.132/24

Software to be installed: kubelet Kube proxy docker flannel etcd

Node02: 192.168.66.133/24

Software to be installed: kubelet Kube proxy docker flannel etcd

2, Environmental preparation

1. Static IP address corresponding to each virtual machine configuration

vi /etc/sysconfig/network-scripts/ifcfg-ens33

2. Prevent IP address change of virtual machine from restarting

systemctl stop NetworkManager
systemctl enable NetworkManager

service network restart   #Restart the network

ping www.baidu.com    #To achieve network communication

3. Do not turn off the firewall.

systemctl start firewalld   #Turn on the firewall
iptables -F    #Clear firewall rules
setenforce 0   #Turn off core protection

3, Deploy ETCD cluster

The communication between etcds is encrypted, so to create a CA certificate, TLS is used to encrypt the communication.

3.1 install cfssl, a certificate making tool

master node:

[root@localhost ~]# mkdir k8s
[root@localhost ~]# cd k8s/ 

//Write the script cfssl.sh, download the certificate making tool cfssl from the official website, and put it directly in the directory / usr/local/bin for the convenience of system identification. Finally, add the execution authority to the tool
[root@localhost k8s]# vi cfssl.sh
curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl
curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson
curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo

 #Execute the script to wait for the download software to be installed
[root@localhost k8s]# bash cfssl.sh  

[root@localhost k8s]# ls /usr/local/bin/ 
#You can see three tools for making certificates
cfssl  cfssl-certinfo  cfssljson

#cfssl: certificate generation tool
#Cfssl certinfo: View Certificate Information
#cfssljson: generate certificate by passing in json file

3.2. Making CA certificate

[root@localhost k8s]# MKDIR etcd Cert / / the location where all certificates are stored
[root@localhost k8s]# MV etcd cert.sh etcd Cert / / the material to generate the certificate
[root@localhost k8s]# cd etcd-cert/

1. Create a profile to generate a ca certificate

cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "www": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]  
      } 
    }
  }
}
EOF

2. Create a signing certificate for the ca certificate

cat > ca-csr.json <<EOF 
{   
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}
EOF

3. CA certificate is generated by Ca signature certificate, and ca-key.pem ca.pem is obtained

cfssl gencert -initca ca-csr.json | cfssljson -bare ca –

4. Specify the communication verification between three nodes of etcd - server-csr.json is required

// Change the IP address to its own node
cat > server-csr.json <<EOF
{
    "CN": "etcd",
    "hosts": [
    "192.168.66.130",
    "192.168.66.132",
    "192.168.66.133"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing"
        }
    ]
}
EOF

5. Using ca-key.pem, ca.pem and server signature certificate to generate ETCD certificate server-key.pem and server.pem

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

3.3 use certificate and etcd script to build etcd cluster

Upload a script etcd.sh that generates the ETCD configuration file to the directory / root/k8s

[root@localhost k8s]# vim etcd.sh 
#!/bin/bash
# example: ./etcd.sh etcd01 192.168.1.10 etcd02=https://192.168.1.11:2380,etcd03=https://192.168.1.12:2380

ETCD_NAME=$1
ETCD_IP=$2
ETCD_CLUSTER=$3

WORK_DIR=/opt/etcd
# Create profile template for node
cat <<EOF >$WORK_DIR/cfg/etcd
#[Member]
ETCD_NAME="${ETCD_NAME}"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://${ETCD_IP}:2380"
ETCD_LISTEN_CLIENT_URLS="https://${ETCD_IP}:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://${ETCD_IP}:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://${ETCD_IP}:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://${ETCD_IP}:2380,${ETCD_CLUSTER}"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF
# Create a startup script template for a node
cat <<EOF >/usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=${WORK_DIR}/cfg/etcd
ExecStart=${WORK_DIR}/bin/etcd \
--name=\${ETCD_NAME} \
--data-dir=\${ETCD_DATA_DIR} \
--listen-peer-urls=\${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=\${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
--advertise-client-urls=\${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=\${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=\${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=\${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=new \
--cert-file=${WORK_DIR}/ssl/server.pem \
--key-file=${WORK_DIR}/ssl/server-key.pem \
--peer-cert-file=${WORK_DIR}/ssl/server.pem \
--peer-key-file=${WORK_DIR}/ssl/server-key.pem \
--trusted-ca-file=${WORK_DIR}/ssl/ca.pem \
--peer-trusted-ca-file=${WORK_DIR}/ssl/ca.pem
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF
# Restart the service and set the start-up mode
systemctl daemon-reload
systemctl enable etcd
systemctl restart etcd

Upload the downloaded three software packages to k8s directory

First extract the etcd package to the current directory, and then create the working directory of the etcd cluster

[root@localhost k8s]# tar zxvf etcd-v3.3.10-linux-amd64.tar.gz / / unzip

[root@localhost k8s]# ls etcd-v3.3.10-linux-amd64
Documentation  etcd  etcdctl  README-etcdctl.md  README.md  READMEv2-etcdctl.md
#Later use the etcd and etcdctl application commands in the source package

[root@localhost k8s]# mkdir /opt/etcd/{cfg,bin,ssl} -p / / configuration file, command file, certificate
[root@localhost k8s]# ls /opt/etcd/
bin  cfg  ssl

1. Put the etcd and etcdctl execution files in / opt/etcd/bin/

[root@localhost k8s]# mv etcd-v3.3.10-linux-amd64/etcd etcd-v3.3.10-linux-amd64/etcdctl /opt/etcd/bin

2. Copy the certificate to / opt/etcd/ssl /

[root@localhost k8s]# cp etcd-cert/*.pem /opt/etcd/ssl/

Execute etcd.sh script to generate configuration script and service startup script of etcd cluster, enter stuck state and wait for other nodes to join

// Pay attention to IP address modification
[root@localhost k8s]#  bash etcd.sh etcd01 192.168.66.130 etcd02=https://192.168.66.132:2380,etcd03=https://192.168.66.133:2380

//Using another session window, you will find that the etcd process has been opened
[root@localhost ~]# ps aux | grep etcd

3.4. node nodes join ETCD cluster (to realize internal communication)

1. Copy the certificate on the master node to other nodes

[root@localhost k8s]# scp -r /opt/etcd/ root@192.168.66.132:/opt

[root@localhost k8s]# scp -r /opt/etcd/ root@192.168.66.133:/opt

2. Copy the startup script of the master node to other nodes

[root@localhost k8s]# scp /usr/lib/systemd/system/etcd.service root@192.168.66.132:/usr/lib/systemd/system

[root@localhost k8s]# scp /usr/lib/systemd/system/etcd.service root@192.168.66.133:/usr/lib/systemd/system

3. Modify the configuration file on node01 node

[root@localhost system]# cd /opt/etcd/cfg/
[root@localhost cfg]# ls
etcd
[root@localhost cfg]# vim etcd

4. Modify the configuration file on node02 node

[root@localhost system]# cd /opt/etcd/cfg/
[root@localhost cfg]# ls
etcd
[root@localhost cfg]# vim etcd

5. Enter bash in the master node and wait for the node to join the cluster

[root@localhost k8s]# bash etcd.sh etcd01 192.168.66.130 etcd02=https://192.168.66.132:2380,etcd03=https://192.168.66.133:2380

6. Start node01 and node02 nodes quickly at the same time

[root@localhost ~]# systemctl start etcd
[root@localhost ~]# systemctl status etcd

3.5 check the cluster status

Execute on the master node. Note: execute the command to check the cluster under etcd Cert /

[root@localhost k8s]# cd etcd-cert/
[root@localhost etcd-cert]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://192.168.66.130:2379,https://192.168.66.132:2379,https://192.168.66.133:2379" cluster-health

4, docker engine deployment

All node nodes must deploy Docker engine. For Docker installation and deployment, please refer to my previous blog: [Docker deployment and image acceleration, network optimization]

5, Deploy the flannel network components

5.1. Establish ETCD cluster and external communication

1. On the master node, write the allocated subnet segment to ETCD for flannel to use

[root@localhost etcd-cert]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://192.168.66.130:2379,https://192.168.66.132:2379,https://192.168.66.133:2379" set /coreos.com/network/config '{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}'

View information written

[root@localhost etcd-cert]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://192.168.66.130:2379,https://192.168.66.132:2379,https://192.168.66.133:2379" get /coreos.com/network/config

2. Two node nodes: upload the software package flannel and extract it to the host directory.

//Copy to all node nodes (just deploy on node nodes)
[root@localhost k8s]# scp flannel-v0.10.0-linux-amd64.tar.gz root@192.168.66.132:/root
[root@localhost k8s]# scp flannel-v0.10.0-linux-amd64.tar.gz root@192.168.66.133:/root

//Decompression of all node operations
[root@localhost ~]# tar zxvf flannel-v0.10.0-linux-amd64.tar.gz 
flanneld
mk-docker-opts.sh
README.md

3. Create k8s working directory on two node nodes

[root@localhost ~]# mkdir /opt/kubernetes/{cfg,bin,ssl} -p
[root@localhost ~]# mv mk-docker-opts.sh flanneld /opt/kubernetes/bin/

Upload the script flannel.sh that can generate the configuration file and startup file.

[root@localhost ~]# vim flannel.sh
#!/bin/bash

ETCD_ENDPOINTS=${1:-"http://127.0.0.1:2379"}

cat <<EOF >/opt/kubernetes/cfg/flanneld

FLANNEL_OPTIONS="--etcd-endpoints=${ETCD_ENDPOINTS} \
-etcd-cafile=/opt/etcd/ssl/ca.pem \
-etcd-certfile=/opt/etcd/ssl/server.pem \
-etcd-keyfile=/opt/etcd/ssl/server-key.pem"

EOF

cat <<EOF >/usr/lib/systemd/system/flanneld.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network-online.target network.target
Before=docker.service

[Service]
Type=notify
EnvironmentFile=/opt/kubernetes/cfg/flanneld
ExecStart=/opt/kubernetes/bin/flanneld --ip-masq \$FLANNEL_OPTIONS
ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
Restart=on-failure

[Install]
WantedBy=multi-user.target

EOF

systemctl daemon-reload
systemctl enable flanneld
systemctl restart flanneld

4. Two node nodes enable the flannel network function

[root@localhost ~]# bash flannel.sh https://192.168.66.130:2379,https://192.168.66.132:2379,https://192.168.66.133:2379

Check whether the network status is running

[root@localhost ~]# systemctl status flanneld

5.2. Configure Docker to connect to flannel network

Two node nodes: modify the configuration file of docker

[root@localhost ~]# vim /usr/lib/systemd/system/docker.service
//Modify and add two places:
EnvironmentFile=/run/flannel/subnet.env
$DOCKER_NETWORK_OPTIONS

View the subnet segments assigned by the flanne network

[root@localhost ~]# cat /run/flannel/subnet.env 

Restart docker service

[root@localhost ~]# systemctl daemon-reload 
[root@localhost ~]# systemctl restart docker

5.3 verification of flannel network interworking

1. The two node nodes create and enter the centos:7 container automatically.

[root@localhost ~]# docker run -it centos:7 /bin/bash

[root@a57795cdc6ef /]# yum install net-tools -y
#You can use ifconfig command after installation

2. ifconfig checks the IP address and uses ping command to check whether the network is interworking

After verification, it can communicate with each other, and the flannel network is completed!