k8s-v1.20.10 1master&2node binary deployment guidance document

k8s-v1.20.10 1master&2node

Experimental environment

Host network information and component information

K8S cluster role	IP	host name	Installed components
master	192.168.0.10	k8s-master-1	apiserver,controller-manager,scheduler,etcd,docker,kubectl,kubelet,kube-proxy,calico,coredns,metric-server
node	192.168.0.11	k8s-node-1	kubelet,kube-proxy,docker,calico
node	192.168.0.12	k8s-node-2	kubelet,kube-proxy,docker,calico

Note: under normal circumstances, the master node is only responsible for scheduling, not running Kube proxy, calico, coredns and metric server. In consideration of saving resources, the master node is also responsible for working here

#System version 
	Centos7.9(4.19.12-1.el7.elrepo.x86_64)

# to configure
	4GB Memory/2vcpu/70G Hard disk, turn on Virtualization,NAT Network mode
	
# Component version
	k8s-server&k8s-node(apiserver,kubectl,kube-scheduler,kube-proxy) 1.20.10
	etcd 3.5.0
	pause: v3.6
	calico/node: v3.20.1
	calico/pod2daemon-flexvol: v3.20.1
	calico/cni: v3.20.1
	coredns/coredns: v1.7.0
	docker: 20.10.8
	metric-server: v0.4.1

# network
	service: 10.0.0.0/24
	pod: 10.70.2.0/24

Host certificate information

There are three sets of Ca institutions: one apiserver, one etcd, and one api aggregation layer (since sharing a CA with apiserver may cause conflicts, a separate CA is used here). The issuing institutions are: Ca apiserver, CA etcd, and front proxy ca

Host initialization

Configure host name

# master-1
	hostnamectl set-hostname k8s-master-1 && bash

# node-1
	hostnamectl set-hostname k8s-node-1 && bash

# node-2
	hostnamectl set-hostname k8s-node-2 && bash

Configure hosts file

# master,node
cat >> /etc/hosts <<EOF
192.168.0.10 k8s-master-1
192.168.0.11 k8s-node-1
192.168.0.12 k8s-node-2
EOF

Secret free login

# master	
	ssh-keygen -t rsa
	ssh-copy-id -i .ssh/id_rsa.pub root@k8s-node-1
	ssh-copy-id -i .ssh/id_rsa.pub root@k8s-node-2

Turn off firewall

# master,node

# Turn off firewall
	systemctl disable firewalld --now

# Close selinux
	sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
	sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux
	setenforce 0

Close swap partition

# master,node
	swapoff -a && sysctl -w vm.swappiness=0
	sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab

Configure yum source

# master,node
	curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
	yum install -y yum-utils device-mapper-persistent-data lvm2
	yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
	sed -i -e '/mirrors.cloud.aliyuncs.com/d' -e '/mirrors.aliyuncs.com/d' /etc/yum.repos.d/CentOS-Base.repo

# Install base dependency package
	yum install -y yum-utils device-mapper-persistent-data lvm2 wget net-tools nfs-utils lrzsz gcc gcc-c++ make cmake libxml2-devel openssl-devel curl curl-devel unzip sudo ntp libaio-devel wget vim ncurses-devel autoconf automake zlib-devel  python-devel epel-release openssh-server socat  ipvsadm conntrack ntpdate

Configure synchronization time

# master,node
# Synchronization time
    yum install ntpdate -y
    ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
    echo 'Asia/Shanghai' >/etc/timezone
    ntpdate time2.aliyun.com
    
# Join crontab
	*/5 * * * * /usr/sbin/ntpdate time2.aliyun.com

Upgrade kernel

# master,node# Update system	yum update -y --exclude=kernel* # take kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm，kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm Upload to the second node	for i in k8s-node-1 k8s-node-2; do scp kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm root@$i:/root ;done	# Install kernel	yum localinstall -y kernel-ml*	# All nodes change the kernel boot order and turn on the kernel user namespace	grub2-set-default  0 && grub2-mkconfig -o /etc/grub2.cfg	grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"	# Restart all nodes and check whether the default kernel is 4.19 	 grubby --default-kernel

Modify kernel parameters

# master,node,# Add the following at the end cat > > / etc / security / limits.conf < < EOF * soft nofile 65536 * hard nofile 131072 * soft nproc 65535 * hard nproc 655350 * soft memlock unlimited * hard memlock unlimited EOF

# master,node# If used firewalld I'm not used to it. I can install it iptables	yum install iptables-services -y	service iptables stop   && systemctl disable iptables	# Install the appropriate package	yum install ipvsadm ipset sysstat conntrack libseccomp -y# If you enable IPVS, iptables will be used for packet forwarding if you do not enable IPVS, but the efficiency is low. Therefore, it is recommended to enable IPVS on the official website. cat > /etc/modules-load.d/ipvs.conf <<EOFip_ vsip_ vs_ lcip_ vs_ wlcip_ vs_ rrip_ vs_ wrrip_ vs_ lblcip_ vs_ lblcrip_ vs_ dhip_ vs_ ship_ vs_ foip_ vs_ nqip_ vs_ sedip_ vs_ ftpip_ vs_ shnf_ conntrackip_ tablesip_ setxt_ setipt_ setipt_ rpfilteript_ REJECTipipEOFsystemctl enable --now systemd-modules-load.service

# master,node# open k8s Kernel parameters cat <<EOF > /etc/sysctl.d/k8s.confnet.ipv4.ip_forward = 1net.bridge.bridge-nf-call-iptables = 1net.bridge.bridge-nf-call-ip6tables = 1fs.may_detach_mounts = 1vm.overcommit_memory=1vm.panic_on_oom=0fs.inotify.max_user_watches=89100fs.file-max=52706963fs.nr_open=52706963net.netfilter.nf_conntrack_max=2310720net.ipv4.tcp_keepalive_time = 600net.ipv4.tcp_keepalive_probes = 3net.ipv4.tcp_keepalive_intvl =15net.ipv4.tcp_max_tw_buckets = 36000net.ipv4.tcp_tw_reuse = 1net.ipv4.tcp_max_orphans = 327680net.ipv4.tcp_orphan_retries = 3net.ipv4.tcp_syncookies = 1net.ipv4.tcp_max_syn_backlog = 16384net.ipv4.ip_conntrack_max = 65536net.ipv4.tcp_max_syn_backlog = 16384net.ipv4.tcp_timestamps = 0net.core.somaxconn = 16384EOFsysctl --system# Check whether the k8s parameter is effective. rebootlsmod | grep --color=auto -e ip_vs -e nf_conntrack

Install docker CE

# master,node# Install docker CE 	 yum install docker-ce.* -ymkdir /etc/dockercat > /etc/docker/daemon.json <<EOF{  "exec-opts": ["native.cgroupdriver=systemd"],  "registry-mirrors": [" https://ornb7jit.mirror.aliyuncs.com "],  "default-ipc-mode": "shareable"}EOFsystemctl daemon-reload && systemctl enable --now docker

Install cfssl

# master Node installation# Process strategy

CA initialization

Note:

All certificates are generated in the master node and then distributed to other node nodes
Etcd, apiserver and apiaggregion use three sets of CA institutions to issue certificates. Generally, etcd, apiserver and other components communicating with apiserver can share one set of CA institutions and apiaggregion one set of CA institutions

# Create a folder for certificates 	 mkdir /root/ssl

# Create CA configuration file cat > ca-config.json < < EOF {"signing": {"default": {"expiry": "87600h"}, "profiles": {"kubernetes": {"expiry": "87600h", "usages": ["signing", "key enclosure", "server auth" ,                    "client auth"                ]            }        }    }}EOF

Notes:

field	explain
signing	Indicates that the certificate can be used to sign other certificates. In the generated ca.pem certificate, CA=TRUE
server auth	Indicates that the client can use the certificate to verify the certificate provided by the server
client auth	Indicates that the server can use the certificate to verify the certificate provided by the client;
config.json	Multiple profiles can be defined to specify different expiration time, usage scenario and other parameters; Use a profile when signing a certificate later

etcd-ca

# Create CA request file cat > etcd-ca-csr.json < < EOF {"CN": "etcd", "key": {"algo": "RSA", "size": 2048}, "names": [{"C": "CN", "L": "Hunan", "ST": "Changsha", "O": "k8s", "Ou": "system"}]} EOF

Notes:

field	explain
hosts	This field is empty. Any host can use etcd-ca.pem
CN	Common Name, Kube apiserver extracts this field from the certificate as the requested user name. The browser uses this field to verify whether the website is legal and the specific website domain name for applying for SSL certificate
C	The country of the applicant can only be a two letter country code. For example, China should fill in CN
L	Locality, region, city
ST	State, state, province
O	Organization, Kube apiserver extracts this field from the certificate as the group and company name to which the requesting user belongs
OU	Department name

# generate CA certificate[root@k8s-master-1 ssl]# cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare etcd-ca2021/10/04 09:32:53 [INFO] generating a new CA key and certificate from CSR2021/10/04 09:32:53 [INFO] generate received request2021/10/04 09:32:53 [INFO] received CSR2021/10/04 09:32:53 [INFO] generating key: rsa-20482021/10/04 09:32:53 [INFO] encoded CSR2021/10/04 09:32:53 [INFO] signed certificate with serial number 465476681475479358683323025732390386015350218727# View generated content[root@k8s-master-1 ssl]# ls etcd*etcd-ca.csr  etcd-ca-csr.json  etcd-ca-key.pem  etcd-ca.pem

Notes:

Private key generated by etcd-ca-key.pem
The certificate generated by etcd-ca.pem will be used later to issue the certificate

kube-apiserver-ca

# establish CA Request file cat > kube-apiserver-ca-csr.json <<EOF {  "CN": "kubernetes",  "key": {      "algo": "rsa",      "size": 2048  },  "names": [{      "C": "CN",      "ST": "hunan",      "L": "changsha",      "O": "k8s",      "OU": "system"    }]}EOF# Generate CA certificate 	 cfssl gencert -initca kube-apiserver-ca-csr.json | cfssljson -bare kube-apiserver-ca

front-proxy-ca

# establish CA Request file cat > front-proxy-ca-csr.json <<EOF {  "CN": "kubernetes",  "key": {      "algo": "rsa",      "size": 2048  },  "names": [{      "C": "CN",      "ST": "hunan",      "L": "changsha",      "O": "k8s",      "OU": "system"    }]}EOF# Generate CA certificate 	 cfssl gencert -initca front-proxy-ca-csr.json | cfssljson -bare front-proxy-ca

Deploy etcd

Kubernetes uses etcd for data storage, so an etcd database should be prepared first. In order to solve the single point of failure of etcd, it should be deployed in a cluster. If three sets are used as a cluster, one fault can be tolerated, and if five sets are used as a cluster, two faults can be tolerated. Since the laboratory is a single node, single node etcd is used here

Create etcd certificate

# establish etcd Request file cat > etcd-csr.json<<EOF {  "CN": "etcd",  "hosts": [    "127.0.0.1",    "192.168.0.10"  ],  "key": {    "algo": "rsa",    "size": 2048  },  "names": [{    "C": "CN",    "ST": "hunan",    "L": "changsha",    "O": "k8s",    "OU": "system"  }]}EOF# Generate certificate	cfssl gencert -ca=etcd-ca.pem -ca-key=etcd-ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson  -bare etcd# be careful: 	 hosts needs to fill in the IP address of the machine running etcd

Create etcd profile

# upload etcd Compressed package	mv etcd etcdctl etcdutl /usr/bin# Create the appropriate folder	mkdir -p /etc/etcd/ssl	mkdir -p /var/lib/etcd/default.etcd# establish etcd configuration file cat > /etc/etcd/etcd.conf<<EOF#[Member]ETCD_DATA_DIR="/var/lib/etcd/default.etcd"ETCD_LISTEN_PEER_URLS="https://192.168.0.10:2380"ETCD_LISTEN_CLIENT_URLS="https://192.168.0.10:2379,http://127.0.0.1:2379"ETCD_NAME="etcd1"#[Clustering]ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.10:2380"ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.10:2379"ETCD_INITIAL_CLUSTER="etcd1=https://192.168.0.10:2380"ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"ETCD_INITIAL_CLUSTER_STATE="new"EOF # create startup service file cat > / usr / lib / SYSTEMd / system / etcd. Service < < EOF "[unit] description = etcd serverafter = network. Targetafter = network online. Targetwants = network online. Target [service] Type=notifyEnvironmentFile=/etc/etcd/etcd.confWorkingDirectory=/var/lib/etcd/ExecStart=/usr/bin/etcd \    --cert-file=/etc/etcd/ssl/etcd.pem \    --key-file=/etc/etcd/ssl/etcd-key.pem \    --trusted-ca-file=/etc/etcd/ssl/etcd-ca.pem \    --peer-cert-file=/etc/etcd/ssl/etcd.pem \    --peer-key-file=/etc/etcd/ssl/etcd-key.pem \    --peer-trusted-ca- File = / etc / etcd / SSL / etcd-ca.pem \ -- peer client cert auth \ -- client cert authrestart = on failurerestartsec = 5limitnofile = 65536 [install] wantedby = multi-user.targeteofsystemctl daemon reload # move the certificate to the corresponding location 	 cp etcd.pem etcd-key.pem etcd-ca.pem /etc/etcd/ssl/ 	#  Start etcd 	 systemctl enable etcd --now 	#  View etcd cluster status[ root@k8s -master-1 ssl] # etcdctl --write-out=table --cacert=/etc/etcd/ssl/etcd-ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints= https://192.168.0.10:2379   endpoint health+---------------------------+--------+------------+-------+|         ENDPOINT          | HEALTH |    TOOK    | ERROR |+---------------------------+--------+------------+ -------+|  https://192.168.0.10:2379  |   true | 6.250211ms |       |+---------------------------+--------+------------+-------+

Notes:

ETCD_DATA_DIR			# Data directory ETCD_LISTEN_PEER_URLS	# Cluster communication listening address ETCD_LISTEN_CLIENT_URLS	# Client access listening address ETCD_NAME				# Node name, unique in the cluster ETCD_INITIAL_ADVERTISE_PEER_URLS 	# Cluster notification address ETCD_ADVERTISE_CLIENT_URLS			# Client notification address ETCD_INITIAL_CLUSTER				# Cluster node address ETCD_INITIAL_CLUSTER_TOKEN			# colony TokenETCD_INITIAL_CLUSTER_STATE			#The current status of joining a cluster. New is a new cluster, and existing means joining an existing cluster

--cert-file :  Client server TLS Path to certificate file,etcd The certificate will be sent to apiserver，Hand over apiserver authentication--key-file: Client server TLS The path of the key file, which will be used to encrypt data for subsequent communication--trusted-ca-file: Path to the client server TLS credible CA Certificate file, apiserver visit etcd The certificate sent is specified by this CA De authentication--peer-key-file: Peer server TLS The path to the key file. This is the key to peer-to-peer traffic for servers and clients--peer-trusted-ca-file: Peer server TLS credible CA Path to file--peer-client-cert-auth: Enable peer client certificate authentication--client-cert-auth: Enable client certificate validation note: normally etcd Should be used server End certificate, apiserver Should use etcd client End certificate. Here, the same certificate is used for later simple maintenance

Deploy apiserver

Upload k8s component

# upload kubernetes-server Binary package(master)	cp kube-apiserver kube-controller-manager kube-scheduler kubectl kubelet kube-proxy /usr/bin	scp kubectl kube-proxy kubelet root@k8s-node-1:/usr/bin	for i in root@k8s-node-1 root@k8s-node-2; do scp kubelet kube-proxy $i:/usr/bin; done# Create related directories (master,node) 	 mkdir -p /etc/kubernetes/ssl 	 mkdir -p /var/log/kubernetes

Create a token.csv file

# Format: token，user name, UID，User group,kubelet-bootstrap This user will be api-server Trusted cat > token.csv << EOF$(head -c 16 /dev/urandom | od -An -t x | tr -d ' '),kubelet-bootstrap,10001,"system:kubelet-bootstrap"EOF# System: kubelet bootstrap is built in this group

Note: the following part of token.csv is used to automatically issue certificates to kubelet

Create apiserver certificate

# establish apiserver Request file cat > kube-apiserver-csr.json <<EOF{  "CN": "kubernetes",  "hosts": [    "127.0.0.1",    "192.168.0.10",    "10.0.0.1",    "kubernetes",    "kubernetes.default",    "kubernetes.default.svc",    "kubernetes.default.svc.cluster",    "kubernetes.default.svc.cluster.local"  ],  "key": {    "algo": "rsa",    "size": 2048  },  "names": [    {      "C": "CN",      "ST": "hunan",      "L": "changsha",      "O": "k8s",      "OU": "system"    }  ]}EOF# host	host Fill in operation apiserver Host IP/VIP,service The first one IP，The rest can be filled in according to the above, node Node is used bootstrap Mechanism automatically issues certificates without IP Fill it in	Normally hosts Field IP For all Master/LB/VIP IP	# Generate certificate 	 cfssl gencert -ca=kube-apiserver-ca.pem -ca-key=kube-apiserver-ca-key.pem -config=ca-config.json -profile=kubernetes kube-apiserver-csr.json | cfssljson  -bare kube-apiserver

Create front proxy CA certificate

# establish apiaggregation Certificate request file cat > front-proxy-client-csr.json <<EOF {  "CN": "front-proxy-client",  "key": {    "algo": "rsa",    "size": 2048  },  "names": [{    "C": "CN",    "ST": "hunan",    "L": "changsha",    "O": "k8s",    "OU": "system"  }]}EOF# Generate certificate 	 cfssl gencert -ca=front-proxy-ca.pem -ca-key=front-proxy-ca-key.pem -config=ca-config.json -profile=kubernetes front-proxy-client-csr.json | cfssljson  -bare front-proxy-client

Create service public / private key

# Generate private key	openssl genrsa -out ./service.key 2048# Generate public key 	 openssl rsa -in ./service.key -pubout -out ./service.pub

Note: this pair of public and private keys is mainly used for service account

Create apiserver configuration file

#  establish apiserver configuration file cat > /usr/lib/systemd/system/kube-apiserver.service <<"EOF"[Unit]Description=Kubernetes API ServerDocumentation=https://github.com/kubernetes/kubernetesAfter=etcd.serviceWants=etcd.service [Service]ExecStart=/usr/bin/kube-apiserver \    --enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds, ResourceQuota \    --anonymous-auth=false \    --bind-address=192.168.0.10 \    --secure-port=6443 \    --advertise-address=192.168.0.10 \    --insecure-port=0 \    --authorization-mode=Node, RBAC \    --runtime-config=api/all=true \    --enable-bootstrap-token-auth \    --token-auth-file=/etc/kubernetes/token.csv \    --service-cluster-ip-range=10.0.0.0/24 \    --service-node-port-range=30000-50000 \    --service-account-key-file=/etc/kubernetes/ssl/service.pub \    --service-account-signing-key-file=/etc/kubernetes/ssl/service.key \     --service-account-issuer= https://kubernetes.default.svc.cluster.local  \    --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem  \    --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \    --client-ca-file=/etc/kubernetes/ssl/kube-apiserver-ca.pem \    --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem \    -- kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem \    --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \    --etcd-certfile=/etc/etcd/ssl/etcd.pem \    --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \    --etcd-servers= https://192.168.0.10:2379  \    --requestheader-client-ca-file=/etc/kubernetes/ssl/front-proxy-ca.pem \    --requestheader-allowed-n ames=front-proxy-client   \    --requestheader-extra-headers-prefix=X-Remote-Extra-  \    --requestheader-group-headers=X-Remote-Group     \    --requestheader-username-headers=X-Remote-User   \    --proxy-client-cert-file=/etc/kubernetes/ssl/front-proxy-client.pem  \    --proxy-client-key-file=/etc/kubernetes/ssl/front-proxy-client-key.pem    \     --enable-swagger-ui=true \    --allow-privileged=true \    --apiserver-count=1 \    --audit-log-maxage=30 \    --audit-log-maxbackup=3 \    --audit-log-maxsize=100 \    --audit-log-path=/var/log/kube-apiserver-audit.log \    --event-ttl=1h \    --alsologtostderr=true \    --logtostderr=false \    --log-dir=/var/log/kubernetes \    --v=2    Resta RT = on failurerestartsec = 5type = notifylimitnofile = 65536 [install] wantedby = multi-user.targeteof # copy the certificate to the corresponding directory 	 cp service.pub service.key kube-apiserver.pem kube-apiserver-key.pem kube-apiserver-ca.pem kube-apiserver-ca-key.pem front-proxy-client.pem front-proxy-client-key.pem front-proxy-ca.pem /etc/kubernetes/ssl/ 	 cp token.csv /etc/kubernetes# startup 	 systemctl daemon-reload  	 systemctl enable kube-apiserver.service --now# check for normal operation 	 Systemctl status Kube apiserver# does not carry access certificate[ root@k8s -master-1 ~]# curl -k  https://192.168.0.10:6443 {  "kind": "Status",  "apiVersion": "v1",  "metadata": {      },  "status": "Failure",  "message": "Unauthorized",  "reason": "Unauthorized",  "code" : 401 # normally # carry the certificate for access. The certificate error is reported here. It is said on the Internet that the combination of names fields in csr is different. No specific reason has been found at present 	 curl -v --cert /etc/kubernetes/ssl/kube-apiserver.pem --key /etc/kubernetes/ssl/kube-apiserver-key.pem --cacert /etc/kubernetes/ssl/kube-apiserver-ca.pem  https://192.168.0.10:6443/healthz

Notes:

--enable-admission-plugins: In addition to the plug-ins enabled by default, additional plug-ins should be started admission plug-in unit--anonymous-auth: Allow anonymous requests to API server The secure port for. Requests that are not rejected by other authentication methods will be treated as anonymous requests system username: anonymous，system group name: unauthenticated. Default: true--bind-address: apiserver Listening address--secure-port: HTTPS Communication port number--advertise-address: Publish to cluster members apiserver of IP Address, which must be accessible by members of the cluster--insecure-port: Whether to use HTTP visit apiserver，0 Default prohibition--authorization-mode: An ordered list of plug-ins that perform authorization on secure ports. Default: AlwaysAllow Comma separated list: AlwaysAllow,AlwaysDeny,ABAC,Webhook,RBAC,Node--runtime-config: api/all=true Enable all apiserver of api--enable-bootstrap-token-auth: Enable allow'kube-system' namespace Medium secrets Type'bootstrap.kubernetes.io/token'be used for TLS Boot authentication--service-cluster-ip-range: CIDR IP Range for assignment service colony IP. Cannot be associated with a node assigned to it pod Any IP Range overlap. Default: 10.0.0.0/24--service-node-port-range: by NodePort Port range reserved by the visibility service. Default: 30000-32767--service-account-key-file: autograph ServiceAccount Token Public key file, kube-controller-manager of --service-account-private-key-file Specifies the private key file, which is used in pairs--service-account-signing-key-file: Point to include service account token The path to the issuer's current private key file. The issuer will sign the issued private key with this private key ID token. Need to set'TokenRequest' feature gate--service-account-issuer: Service account token Identifier of the publisher. The publisher will“ iss"Declare the identifier. The value is a string or URL--token-auth-file: If set, the file will be used through token Authentication to protect API Secure port of the server--tls-cert-file: whether apiserver As server or client,apiserver Send this certificate to kubelet,etcd,proxy External components--tls-private-key-file: Private key used when accessing--client-ca-file: verification client (kue-controller-manager,kube-scheduler,kubelet,kube-proxy etc.)Requested certificate--kubelet-client-certificate apiserver visit kubelet Certificate and private key used when--kubelet-client-key: apiserver visit kubelet Use the private key of the certificate when--etcd-cafile: award etcd Certificated ca Certificate, subsequent apiserver Use this certificate to authenticate etcd Certificate from--etcd-certfile: apiserver meeting etcd This certificate will be sent to when communicating etcd authentication--etcd-keyfile: stay apiserver And etcd Before negotiating the symmetric secret key encryption information, apiserver This private key will be used to encrypt the information--etcd-servers: etcd IP Addresses, if more than one, separated by commas--requestheader-client-ca-file: This file is used for authentication proxy Certificate from--requestheader-allowed-names: Clients allowed to access common names List. Client common names Your name needs to be in client-ca-file When it is set to a null value, it means that any client can access it--requestheader-extra-headers-prefix: List of request header prefixes to check. Recommended setting is X-Remote-Extra.--requestheader-group-headers: The group name to be checked in the request header--requestheader-username-headers: User name to be checked in the request header--proxy-client-cert-file: apiserver visit Aggregator Certificate of--proxy-client-key-file: apiserver visit Aggregatior Private key used--enable-swagger-ui: apiserver Enable swagger-ui--allow-privileged: If yes true，Container that allows privileged mode. Default: false--apiserver-count: Running in a cluster apiserver Must be a positive number. Default value: 1 when used--endpoint-reconciler-type=master-count Enabled when--audit-log-maxage: The maximum number of days to save the audit log file according to the encoded timestamp in the file name--audit-log-maxbackup: Maximum number of audit log files saved--audit-log-maxsize: Maximum size of audit log file before flow(In megabytes)--audit-log-path: If set, all to apiserver All requests will be recorded in this file.'-'Indicates writing to standard output.--event-ttl: Retention time. Default: 1 h0m0s--alsologtostderr: All log Output to standard error output--logtostderr: The log is written to the label error output instead of the file. The default value is true--log-dir: If it is not empty, write the log file to the directory--v: Log level of the log

Deploy kubectl

Create kubctl certificate

# establish kubectl Certificate request file cat > admin-csr.json <<EOF{  "CN": "admin",  "hosts": [],  "key": {    "algo": "rsa",    "size": 2048  },  "names": [    {      "C": "CN",      "ST": "hunan",      "L": "changsha",      "O": "system:masters",                   "OU": "system"    }  ]}EOF# Generate certificate	cfssl gencert -ca=kube-apiserver-ca.pem -ca-key=kube-apiserver-ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin# Place the certificate in the appropriate location 	 cp admin*.pem /etc/kubernetes/ssl/

Notes:

Cluster admin (built-in Role with maximum permission) binds Group system:masters to Role cluster admin, which grants the permission to call all API s of Kube apiserver
O specify that the Group of the certificate is system:masters, which must be system:masters. Otherwise, kubectl create clusterrolebinding will report an error

Create kubeconfig configuration file for kubectl

# Set cluster parameters	kubectl config set-cluster kubernetes --certificate-authority=kube-apiserver.pem --embed-certs=true --server=https://192.168.0.10:6443 --kubeconfig=kube.config # set client authentication parameters 	 Kubectl config set credentials admin -- client certificate = admin.pem -- client key = admin key. PEM -- embedded certs = true -- kubeconfig = kube.config # set context parameters 	 kubectl config set-context kubernetes --cluster=kubernetes --user=admin --kubeconfig=kube.config 	#  Set default context 	 Kubectl config use context kubernetes -- kubeconfig = kube.config # copy to the specified directory MKDIR - P ~ /. Kube CP - I kube.config ~ /. Kube / config 	#  View SVC[ root@k8s -Master-1 ssl]# kubectl get svcname type cluster-ip external-ip port (s) agekubernetes clusterip 10.0.0.1 < none > 443 / TCP 52m# authorizes apiserver user to access kubelet. This user declares in the CN field of apiserver certificate that subsequent apiservers need to communicate with kubelet 	 kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes

Deploy Kube Controller Manager

Create Kube controller manager certificate

# establish kube-controller-manager Certificate request file cat > kube-controller-manager-csr.json <<EOF{    "CN": "system:kube-controller-manager",    "key": {        "algo": "rsa",        "size": 2048    },    "hosts": [      "127.0.0.1",      "192.168.0.10"    ],    "names": [      {        "C": "CN",        "ST": "hunan",        "L": "changsha",        "O": "system:kube-controller-manager",        "OU": "system"      }    ]}EOF# Generate certificate 	 cfssl gencert -ca=kube-apiserver-ca.pem -ca-key=kube-apiserver-ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

Notes:

System: Kube controller manager, the built-in clusterrolebindings system of kubernetes: Kube controller manager gives Kube controller manager the necessary permissions to work

Create kubeconfig for Kube Controller Manager

# Set cluster parameters	kubectl config set-cluster kubernetes --certificate-authority=kube-apiserver-ca.pem --embed-certs=true --server=https://192.168.0.10:6443 -- kubeconfig = Kube controller manager.kubeconfig# set client authentication parameters 	 kubectl config set-credentials system:kube-controller-manager --client-certificate=kube-controller-manager.pem --client-key=kube-controller-manager-key.pem --embed-certs=true --kubeconfig=kube-controller-manager.kubeconfig 	#  Setting context parameters 	 Kubectl config set context system: Kube Controller Manager -- cluster = kubernetes -- user = system: Kube Controller Manager -- kubeconfig = Kube controller manager.kubeconfig# sets the default context 	 kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig

Create Kube controller manager configuration file

# establish kube-controller-manager Launch profile cat > /usr/lib/systemd/system/kube-controller-manager.service <<"EOF"[Unit]                                                                     Description=Kubernetes Controller ManagerDocumentation=https://github.com/kubernetes/kubernetes[Service]       ExecStart=/usr/bin/kube-controller-manager \    --port=10252 \    --secure-port=10257 \    --bind-address=127.0.0.1 \    --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \    --service-cluster-ip-range=10.0.0.0/24 \    --cluster-name=kubernetes \    --cluster-signing-cert-file=/etc/kubernetes/ssl/kube-apiserver-ca.pem \    --c luster-signing-key-file=/etc/kubernetes/ssl/kube-apiserver-ca-key.pem \    --cluster-signing-duration=87600h \    --allocate-node-cidrs=true \    --cluster-cidr=10.70.2.0/24 \    --root-ca-file=/etc/kubernetes/ssl/kube-apiserver-ca.pem \    --service-account-private-key-file=/etc/kubernetes/ssl/service.key \    --use-service-account-credentials=tr ue \    --leader-elect=true \    --feature-gates=RotateKubeletServerCertificate=true,RotateKubeletClientCertificate=true \    --controllers=*,bootstrapsigner,tokencleaner \    --tls-cert-file=/etc/kubernetes/ssl/kube-controller-manager.pem \    --tls-private-key-file=/etc/kubernetes/ssl/kube-controller-manager-key.pem \ 	-- requestheader-client-ca-file=/etc/kubernetes/ssl/front-proxy-ca.pem \ 	-- requestheader-allowed-names=front-proxy-client   \ 	-- requestheader-extra-headers-prefix=X-Remote-Extra-  \ 	-- requestheader-group-headers=X-Remote-Group     \ 	-- requestheader-username-headers=X-Remote-User   \ 	-- Horizontal pod autoscaler use rest clients = true \ -- alsologtostdir = true \ -- logtostderr = false \ -- log dir = / var / log / kubernetes \ -- v = 2 restart = on failurerestartsec = 5 [install] wantedby = multi-user.targeteof# copy files 	 cp kube-controller-manager*.pem /etc/kubernetes/ssl/ 	 cp kube-controller-manager.kubeconfig /etc/kubernetes / # start the service 	 systemctl daemon-reload 	 Systemctl enable Kube Controller Manager -- now# check the running status of Kube Controller Manager[ root@k8s-master-1 ssl]# kubectl get csWarning: v1 ComponentStatus is deprecated in v1.19+NAME                 STATUS      MESSAGE                                                                                       ERRORscheduler            Unhealthy   Get "http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection refused   controller-manager   Healthy     ok                                                                                            etcd-0               Healthy     {"health":"true","reason":""}

Notes:

--port: be used for HTTP Communication, the default port is 10252, kubectl get cs The default is to use HTTP Mode query kube-controller-manager Component status--secure-port: be used for HTTPS signal communication--bind-address: Because only and apiserver For communication, the listening address can be set to 127.0.0.1--kubeconfig: appoint kubeconfig File path, kube-controller-manager Use it to connect and verify kube-apiserver,It contains CA certificate--service-cluster-ip-range: CIDR IP Range for assignment service colony IP. Cannot be associated with a node assigned to it pod****Any IP Range overlap. Default: 10.0.0.0/24--cluster-name: Prefix of cluster instance. Default value:"kubernetes"--cluster-signing-cert-file: because bootstrap The actual issuing component of the certificate is kube-controller-manager,kubelet To be with apiserver Mutual trust, so you need to use and apiserver same CA Institutions, kube-controller-manager This will be used ca Certificate for kubelet Issue certificate--cluster-signing-key-file: Private key--cluster-signing-duration: bootstrap Issued certificate, the default time is one year--allocate-node-cidrs: Should it be assigned and set up on the cloud provider Pod of CIDR--cluster-cidr: In cluster Pods of CIDR Scope. Requirements --allocate-node-cidrs Flag is true--root-ca-file: If this flag is not empty, the token in the service account will be displayed Secret This root certificate authority will be included in the. The specified flag value must be a legal one PEM Coded CA Certificate package--service-account-private-key-file: And apiserver Medium–service-account-key-file Public private key pair for pairing--use-service-account-credentials: When this flag is true The service account credentials are used separately for each controller--leader-elect: Start the leadership election before executing the main cycle( Leader Election)Client and try to gain leadership. Enabling this flag when running multi copy components helps improve availability--feature-gates: a set key=value Yes, it's used to describe testability/Characteristic gating of experimental functions( Feature Gate)，RotateKubeletServerCertificate=true Option, then kubelet A certificate is automatically initiated when the certificate is about to expire renew Own certificate CSR Request; meanwhile controller manager It needs to be added at startup --feature-gates=RotateKubeletServerCertificate=true Parameters, and then cooperate with the corresponding created parameters ClusterRoleBinding，kubelet client and kubelet server The certificate will be automatically signed--controllers: List of controllers to enable.\* Indicates that all default enabled controllers are enabled; foo Enable named foo Controller of; -foo Indicates that the name is disabled foo The default disabled controllers are: bootstrapsigner and tokencleaner，So you need to add--tls-*-file: kube-controller-manager visit apiserver The certificate and secret key used`Must be with apiserver Same CA Certificate issued by authority`--requestheader*: reference apiserver explain--alsologtostderr: While outputting logs to files, the logs are also written to standard output--log-dir: Log storage path--v2: Logging level

Deploy Kube scheduler

Create Kube scheduler certificate

# Create certificate request file cat > kube-scheduler-csr.json <<EOF{    "CN": "system:kube-scheduler",    "hosts": [      "127.0.0.1",      "192.168.0.10"    ],    "key": {        "algo": "rsa",        "size": 2048    },    "names": [      {        "C": "CN",        "ST": "hunan",        "L": "changsha",        "O": "system:kube-scheduler",        "OU": "system"      }    ]}EOF Note:	O by system:kube-scheduler，kubernetes Built in ClusterRoleBindings system:kube-scheduler Will give kube-scheduler Permissions required for work	# Generate certificate 	 cfssl gencert -ca=kube-apiserver-ca.pem -ca-key=kube-apiserver-ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler

Create kubeconfig for Kube scheduler

# Set cluster parameters	kubectl config set-cluster kubernetes --certificate-authority=kube-apiserver-ca.pem --embed-certs=true --server=https://192.168.0.10:6443 --kubeconfig=kube-scheduler.kubeconfig 	#  Setting client authentication parameters 	 Kubectl config set credentials system: Kube scheduler -- client certificate = Kube scheduler.pem -- client key = Kube scheduler key. PEM -- embedded certs = true -- kubeconfig = Kube scheduler.kubeconfig# set context parameters 	 Kubectl config set context system: Kube scheduler -- cluster = kubernetes -- user = system: Kube scheduler -- kubeconfig = Kube scheduler.kubeconfig# sets the default context 	 kubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig

Create Kube scheduler configuration file

cat > /usr/lib/systemd/system/kube-scheduler.service <<"EOF"[Unit]                                      Description=Kubernetes SchedulerDocumentation=https://github.com/kubernetes/kubernetes[Service]ExecStart=/usr/bin/kube-scheduler  \    --address=127.0.0.1  \    --port=10251  \    --secure-port=10259  \    --kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig  \    --leader-elect=true  \    --alsologtostderr=true  \    --logtostderr=false  \    --log-dir=/var/log/kubernetes  \     --v=2 Restart=on-failureRestartSec=5 [Install]WantedBy=multi-user.targetEOF# copy file 	 cp kube-scheduler*.pem /etc/kubernetes/ssl/ 	 CP Kube scheduler.kubeconfig / etc / kubernetes / # start service 	 systemctl daemon-reload 	 systemctl enable kube-scheduler.service --now# view service status[ root@k8s-master-1 ssl]# kubectl get csWarning: v1 ComponentStatus is deprecated in v1.19+NAME                 STATUS    MESSAGE                         ERRORscheduler            Healthy   ok                              controller-manager   Healthy   ok                              etcd-0               Healthy   {"health":"true","reason":""}

Deploy kubelet

be careful:

In this article, because the master node needs to run calico, coredns and other system components (running in pod mode), the master node needs to deploy kubelet and Kube proxy

# Intercept tokenBOOTSTRAP_TOKEN=$(awk -F "," '{print  }' /etc/kubernetes/token.csv)

Create kubeconfig for kubelet

# Set cluster parameters	kubectl config set-cluster kubernetes --certificate-authority=kube-apiserver-ca.pem --embed-certs=true --server=https://192.168.0.10:6443 -- kubeconfig = kubelet bootstrap.kubeconfig# set client authentication parameters 	 Kubectl config set credentials kubelet bootstrap -- token = ${bootstrap_token} -- kubeconfig = kubelet bootstrap.kubeconfig# set context parameters 	 Kubectl config set context default -- cluster = kubernetes -- user = kubelet bootstrap -- kubeconfig = kubelet bootstrap.kubeconfig# sets the default context 	 kubectl config use-context default --kubeconfig=kubelet-bootstrap.kubeconfig

Create kublet's profile

Note:

"cgroupDriver": "systemd" should be consistent with the docker driver.
Address is replaced by the IP address of the node running kubelet.

# k8s-master-1 node cat > k8s-master-1-kubelet.json <<EOF{  "kind": "KubeletConfiguration",  "apiVersion": "kubelet.config.k8s.io/v1beta1",  "authentication": {    "x509": {      "clientCAFile": "/etc/kubernetes/ssl/kube-apiserver-ca.pem"    },    "webhook": {      "enabled": true,      "cacheTTL": "2m0s"    },    "anonymous": {      "enabled": false    }  },  "authorization": {    "mode": "Webhook",    "webhook": {      "cacheAuthorizedTTL": "5m0s",      "cacheUnauthorizedTTL": "30s"    }  },  "address": "192.168.0.10",  "port": 10250,  "readOnlyPort": 10255,  "cgroupDriver": "systemd",  "hairpinMode": "promiscuous-bridge",  "serializeImagePulls": false,  "featureGates": {    "RotateKubeletClientCertificate": true,    "RotateKubeletServerCertificate": true  },  "clusterDomain": "cluster.local.",  "clusterDNS": ["10.0.0.10"]}EOF# k8s-node-1 node cat > k8s-node-1-kubelet.json <<EOF{  "kind": "KubeletConfiguration",  "apiVersion": "kubelet.config.k8s.io/v1beta1",  "authentication": {    "x509": {      "clientCAFile": "/etc/kubernetes/ssl/kube-apiserver-ca.pem"    },    "webhook": {      "enabled": true,      "cacheTTL": "2m0s"    },    "anonymous": {      "enabled": false    }  },  "authorization": {    "mode": "Webhook",    "webhook": {      "cacheAuthorizedTTL": "5m0s",      "cacheUnauthorizedTTL": "30s"    }  },  "address": "192.168.0.11",  "port": 10250,  "readOnlyPort": 10255,  "cgroupDriver": "systemd",  "hairpinMode": "promiscuous-bridge",  "serializeImagePulls": false,  "featureGates": {    "RotateKubeletClientCertificate": true,    "RotateKubeletServerCertificate": true  },  "clusterDomain": "cluster.local.",  "clusterDNS": ["10.0.0.10"]}EOF# K8s-node-2 node cat > k8s-node-2-kubelet.json < < EOF {"kind": "kubeletconfiguration", "apiversion": "kubelet. Config. K8s. IO / v1beta1", "authentication": {"x509": {"clientcafile": "/ etc / kubernetes / SSL / Kube apiserver ca.pem"}, "webhook": {"enabled": true, "cachettl": "2m0s"}, "anonymous": {"enabled" : false    }  },  "authorization": {    "mode": "Webhook",    "webhook": {      "cacheAuthorizedTTL": "5m0s",      "cacheUnauthorizedTTL": "30s"    }  },  "address": "192.168.0.12",  "port": 10250,  "readOnlyPort": 10255,  "cgroupDriver": "systemd",  "hairpinMode": "promiscuous-bridge",  "serializeImagePulls": false,  "featureGates": {     "RotateKubeletClientCertificate": true,    "RotateKubeletServerCertificate": true  },  "clusterDomain": "cluster.local.",  "clusterDNS": ["10.0.0.10"]}EOF

Create kubelet startup file

# establish kubelet Start configuration,master node cat > kubelet.service<<"EOF"[Unit]Description=Kubernetes KubeletDocumentation=https://github.com/kubernetes/kubernetesAfter=docker.serviceRequires=docker.service[Service] WorkingDirectory=/var/lib/kubeletExecStart=/usr/bin/kubelet \    --bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig \    --cert-dir=/etc/kubernetes/ssl \    --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \    --config=/etc/kubernetes/kubelet.json \    --network-plugin=cni \    --pod-infra-container-image=registry.aliyuncs.com/goo Gle_containers / pause: 3.6 \ -- feature gates = rotatekubeletclientcertificate = true, rotatekubeletservercertificate = true \ -- rotate certificates = true \ -- alsologtosterror = true \ -- logtostderr = false \ -- log dir = / var / log / kubernetes \ -- v = 2restart = on failurerestartsec = 5 [install] wantedby = multi-user.targeteof# move related files 	 mkdir -p /var/lib/kubelet 	 cp kubelet.service /usr/lib/systemd/system/ 	 cp k8s-master-1-kubelet.json /etc/kubernetes/kubelet.json 	 cp kubelet-bootstrap.kubeconfig /etc/kubernetes / # start the service 	 systemctl daemon-reload 	 systemctl enable kubelet --now

Create RBAC rules to automatically approve CSR

apiserver automatically creates two clusterroles:

system:certificates.k8s.io:certificatesigningrequests:nodeclient
system:certificates.k8s.io:certificatesigningrequests:selfnodeclient

# Let's add another one cat <<EOF | kubectl apply -f -kind: ClusterRoleapiVersion: rbac.authorization.k8s.io/v1metadata:  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserverrules:- apiGroups: ["certificates.k8s.io"]  resources: ["certificatesigningrequests/selfnodeserver"]  verbs: ["create"]EOF# take ClusterRole Bind to the appropriate user group to complete automatic approval CSR Request, here system:bootstrappers Group and token.csv Group correspondence in# token.csv，format Token,user name,UID,User group fbecd7fb7d3c75efc7f8bd8c0896addf,kubelet-bootstrap,10001,"system:kubelet-bootstrap"# allow system:bootstrappers Group user creation CSR request	kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --group=system:kubelet-bootstrap# Automatic approval system:bootstrappers Group user TLS bootstrapping First application for certificate CSR Request, clusterrolebinding kubelet-bootstrap and node-client-auto-approve-csr Medium--group=system:kubelet-bootstrap Can be replaced with--user=kubelet-bootstrap,And token.csv bring into correspondence with	kubectl create clusterrolebinding node-client-auto-approve-csr --clusterrole=system:certificates.k8s.io:certificatesigningrequests:nodeclient --group=system:kubelet-bootstrap# Automatic approval system:nodes Group user update kubelet Self and apiserver Of Communication Certificate CSR request	kubectl create clusterrolebinding node-client-auto-renew-crt --clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeclient --group=system:nodes# Automatic approval system:nodes Group user update kubelet 10250 api Port certificate CSR request	kubectl create clusterrolebinding node-server-auto-renew-crt --clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeserver --group=system:nodes	# see csr,Can be found master After the node joins the cluster, the certificate is automatically issued[root@k8s-master-1 ~]# kubectl get csrNAME                                                   AGE     SIGNERNAME                                    REQUESTOR           CONDITIONnode-csr-7yhdBfn1JE3dUOPfvRLVkRzlljdgno9X0C_X0gqipzg   2m16s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Approved,Issued# View node status[root@k8s-master-1 ~]# kubectl get nodesNAME           STATUS     ROLES    AGE    VERSIONk8s-master-1   NotReady   <none>   114s   v1.20.10

Node deployment Kubelet

# Issue relevant certificates to node node		for i in root@k8s-node-1 root@k8s-node-2; do scp kube-apiserver-ca.pem $i:/etc/kubernetes/ssl;scp kubelet.service $i:/usr/lib/systemd/system; scp kubelet-bootstrap.kubeconfig $i:/etc/kubernetes; done		scp k8s-node-1-kubelet.json root@k8s-node-1:/etc/kubernetes/kubelet.json	scp k8s-node-2-kubelet.json root@k8s-node-2:/etc/kubernetes/kubelet.json# node Node start service	for i in root@k8s-node-1 root@k8s-node-2; do ssh $i "mkdir -p /etc/kubernetes/ssl;mkdir -p /var/lib/kubelet; mkdir -p /var/log/kubernetes;systemctl daemon-reload; systemctl enable kubelet --now;"; done	# see node[root@k8s-master-1 ssl]# kubectl get nodesNAME           STATUS     ROLES    AGE   VERSIONk8s-master-1   NotReady   <none>   46m   v1.20.10k8s-node-1     NotReady   <none>   13m   v1.20.10k8s-node-2     NotReady   <none>   13m   v1.20.10

Deploy Kube proxy

Create Kube proxy certificate

# Create certificate request file cat > kube-proxy-csr.json <<EOF{  "CN": "system:kube-proxy",  "key": {    "algo": "rsa",    "size": 2048  },  "names": [{      "C": "CN",      "ST": "hunan",      "L": "changsha",      "O": "system:kube-proxy",      "OU": "system"}]}EOF# Generate certificate 	 cfssl gencert -ca=kube-apiserver-ca.pem -ca-key=kube-apiserver-ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

Notes:

CN: specify the User of the certificate as system: Kube proxy
The predefined rolebinding system: node proxy binds user system: Kube proxy to Role system: node proxy, which grants the right to call the API related to Kube apiserver proxy
The certificate will only be used by Kube proxy as a client certificate, so the hosts field is empty

Create kubeconfig for Kube proxy

# Set cluster parameters	kubectl config set-cluster kubernetes --certificate-authority=kube-apiserver-ca.pem --embed-certs=true --server=https://192.168.0.10:6443 -- kubeconfig = Kube proxy.kubeconfig# set client authentication parameters 	 Kubectl config set credentials Kube proxy -- client certificate = Kube proxy. PEM -- client key = Kube proxy key. PEM -- embedded certs = true -- kubeconfig = Kube proxy.kubeconfig# set context parameters 	 Kubectl config set context default -- cluster = kubernetes -- user = Kube proxy -- kubeconfig = Kube proxy.kubeconfig# sets the default context 	 kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

Create Kube proxy profile

# IP Change to operation kube-proxy node IP that will do# establish k8s-master-1 configuration file cat > k8s-master-1-kube-proxy.yaml << EOFapiVersion: kubeproxy.config.k8s.io/v1alpha1bindAddress: 192.168.0.10clientConnection:  kubeconfig: /etc/kubernetes/kube-proxy.kubeconfigclusterCIDR: 192.168.0.0/24healthzBindAddress: 192.168.0.10:10256kind: KubeProxyConfigurationmetricsBindAddress: 192.168.0.10:10249mode: "ipvs"EOF# establish k8s-node-1 configuration file cat > k8s-node-1-kube-proxy.yaml << EOFapiVersion: kubeproxy.config.k8s.io/v1alpha1bindAddress: 192.168.0.11clientConnection:  kubeconfig: /etc/kubernetes/kube-proxy.kubeconfigclusterCIDR: 192.168.0.0/24healthzBindAddress: 192.168.0.11:10256kind: KubeProxyConfigurationmetricsBindAddress: 192.168.0.11:10249mode: "ipvs"EOF# Create k8s-node-2 configuration file cat > k8s-node-2-kube-proxy.yaml < < eofapiversion: kubeproxy.config.k8s.io/v1alpha1bindaddress: 192.168.0.12clientconnection: kubeconfig: / etc / kubernetes / kube-proxy.kubeconfigugclustercidr: 192.168.0.0/24healthzbindaddress: 192.168.0.12:10256kind: kubeproxyconfigurationmetrics bindaddress: 192.168.0.12:10249mode: "IPVS" EOF

Create Kube proxy startup file

# establish kube-proxy Startup file cat > kube-proxy.service <<"EOF"[Unit]Description=Kubernetes Kube-Proxy ServerDocumentation=https://github.com/kubernetes/kubernetesAfter=network.target [Service]WorkingDirectory=/var/lib/kube-proxyExecStart=/usr/bin/kube-proxy \  --config=/etc/kubernetes/kube-proxy.yaml \  --alsologtostderr=true \  --logtostderr=false \  --log-dir=/var/log/kubernetes \  --v=2Restart=on-failureRestartSec=5LimitNOFILE=65536 [Install]WantedBy=multi-user.targetEOF# copy the certificate and create the relevant folder 	 mkdir -p /var/lib/kube-proxy 	 cp kube-proxy.service /usr/lib/systemd/system/ 	 cp k8s-master-1-kube-proxy.yaml /etc/kubernetes/kube-proxy.yaml 	 CP Kube proxy.kubeconfig / etc / kubernetes / # start service 	 systemctl daemon-reload 	 systemctl enable kube-proxy.service --now

Node deployment Kube proxy

scp k8s-node-1-kube-proxy.yaml root@k8s-node-1:/etc/kubernetes/kube-proxy.yamlscp k8s-node-2-kube-proxy.yaml root@k8s-node-2:/etc/kubernetes/kube-proxy.yamlfor i in root@k8s-node-1 root@k8s-node-2; do scp kube-proxy.service $i:/usr/lib/systemd/system/; scp kube-proxy.kubeconfig $i:/etc/kubernetes; ssh $i "mkdir -p /var/lib/kube-proxy; systemctl daemon-reload; systemctl enable kube-proxy.service --now"; done

Add cluster role

# View the current cluster status. The default should be NotReady，Here, because I deployed calico Yes, so it shows Ready Now you can see the cluster ROLES by none[root@k8s-master-1 ssl]# kubectl get nodesNAME           STATUS   ROLES    AGE   VERSIONk8s-master-1   Ready    <none>   15h   v1.20.10k8s-node-1     Ready    <none>   15h   v1.20.10k8s-node-2     Ready    <none>   15h   v1.20.10# set up k8s-master-1 by master node	kubectl label nodes k8s-master-1 node-role.kubernetes.io/master=# set up k8s-node-*by work node	kubectl label nodes k8s-node-1 node-role.kubernetes.io/node=	kubectl label nodes k8s-node-2 node-role.kubernetes.io/node=# set up master Generally, scheduling is not accepted, but only the scheduling of necessary components	kubectl taint nodes k8s-master-1 node-role.kubernetes.io/master=true:NoSchedule	# Or set master Nodes can also accept scheduling	kubectl taint nodes k8s-master-1 node-role.kubernetes.io/master-# View the current status of the cluster[root@k8s-master-1 ssl]# kubectl get nodesNAME           STATUS   ROLES    AGE   VERSIONk8s-master-1   Ready    master   15h   v1.20.10k8s-node-1     Ready    node     15h   v1.20.10k8s-node-2     Ready    node     15h   v1.20.10

Deploy calico

# Download File	curl -O https://docs.projectcalico.org/manifests/calico.yaml# CALICO_ IPV4POOL_ Where CIDR is changed to pod IP # modification 	-  name: CALICO_IPV4POOL_CIDR 	   value: "10.70.2.0/24"       	-  name: IP_AUTODETECTION_METHOD 	   value: interface=ens33 	  #  Run calico 	 kubectl apply -f calico.yaml

[root@k8s-master-1 ssl]# kubectl get pods -A -o wideNAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE     IP             NODE           NOMINATED NODE   READINESS GATESkube-system   calico-kube-controllers-855445d444-jvxt8   1/1     Running   0          3m22s   10.70.2.65     k8s-node-2     <none>           <none>kube-system   calico-node-44f94                          1/1     Running   0          3m23s   192.168.0.12   k8s-node-2     <none>           <none>kube-system   calico-node-bvpdd                          1/1     Running   0          3m23s   192.168.0.11   k8s-node-1     <none>           <none>kube-system   calico-node-g5p8g                          1/1     Running   0          3m23s   192.168.0.10   k8s-master-1   <none>           <none># see master Node routing, running calico before[root@k8s-master-1 ssl]# route -nKernel IP routing tableDestination     Gateway         Genmask         Flags Metric Ref    Use Iface0.0.0.0         192.168.0.2     0.0.0.0         UG    100    0        0 ens33172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0192.168.0.0     0.0.0.0         255.255.255.0   U     100    0        0 ens33# see master node ipvs Information, running calico before[root@k8s-master-1 ssl]# ipvsadm -lnIP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port Scheduler Flags  -> RemoteAddress:Port           Forward Weight ActiveConn InActConnTCP  10.0.0.1:443 rr  -> 192.168.0.10:6443            Masq    1      0          0   # see master Node routing, running calico after[root@k8s-master-1 ssl]# route -nKernel IP routing tableDestination     Gateway         Genmask         Flags Metric Ref    Use Iface0.0.0.0         192.168.0.2     0.0.0.0         UG    100    0        0 ens3310.70.2.0       192.168.0.11    255.255.255.192 UG    0      0        0 tunl010.70.2.64      192.168.0.12    255.255.255.192 UG    0      0        0 tunl010.70.2.128     0.0.0.0         255.255.255.192 U     0      0        0 *172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0192.168.0.0     0.0.0.0         255.255.255.0   U     100    0        0 ens33# Notes: 	 It can be seen from the above that calico is a node node that can communicate pod across hosts. It is necessary to run calico components on each node to generate corresponding routing information on the host. When it is necessary to access pod across hosts, the traffic will be routed

Deploy coredns

cat > coredns.yaml <<EOFapiVersion: v1kind: ServiceAccountmetadata:  name: coredns  namespace: kube-system---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata:  labels:    kubernetes.io/bootstrapping: rbac-defaults  name: system:corednsrules:  - apiGroups:    - ""    resources:    - endpoints    - services    - pods    - namespaces    verbs:    - list    - watch  - apiGroups:    - discovery.k8s.io    resources:    - endpointslices    verbs:    - list    - watch---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata:  annotations:    rbac.authorization.kubernetes.io/autoupdate: "true"  labels:    kubernetes.io/bootstrapping: rbac-defaults  name: system:corednsroleRef:  apiGroup: rbac.authorization.k8s.io  kind: ClusterRole  name: system:corednssubjects:- kind: ServiceAccount  name: coredns  namespace: kube-system---apiVersion: v1kind: ConfigMapmetadata:  name: coredns  namespace: kube-systemdata:  Corefile: |    .:53 {        errors        health {          lameduck 5s        }        ready        kubernetes cluster.local in-addr.arpa ip6.arpa {          fallthrough in-addr.arpa ip6.arpa        }        prometheus :9153        forward . /etc/resolv.conf {          max_concurrent 1000        }        cache 30        loop        reload        loadbalance    }---apiVersion: apps/v1kind: Deploymentmetadata:  name: coredns  namespace: kube-system  labels:    k8s-app: kube-dns    kubernetes.io/name: "CoreDNS"spec:  # replicas: not specified here:  # 1. Default is 1.  # 2. Will be tuned in real time if DNS horizontal auto-scaling is turned on.  strategy:    type: RollingUpdate    rollingUpdate:      maxUnavailable: 1  selector:    matchLabels:      k8s-app: kube-dns  template:    metadata:      labels:        k8s-app: kube-dns    spec:      priorityClassName: system-cluster-critical      serviceAccountName: coredns      tolerations:        - key: "CriticalAddonsOnly"          operator: "Exists"      nodeSelector:        kubernetes.io/os: linux      affinity:         podAntiAffinity:           preferredDuringSchedulingIgnoredDuringExecution:           - weight: 100             podAffinityTerm:               labelSelector:                 matchExpressions:                   - key: k8s-app                     operator: In                     values: ["kube-dns"]               topologyKey: kubernetes.io/hostname      containers:      - name: coredns        image: coredns/coredns:1.7.0        imagePullPolicy: IfNotPresent        resources:          limits:            memory: 170Mi          requests:            cpu: 100m            memory: 70Mi        args: [ "-conf", "/etc/coredns/Corefile" ]        volumeMounts:        - name: config-volume          mountPath: /etc/coredns          readOnly: true        ports:        - containerPort: 53          name: dns          protocol: UDP        - containerPort: 53          name: dns-tcp          protocol: TCP        - containerPort: 9153          name: metrics          protocol: TCP        securityContext:          allowPrivilegeEscalation: false          capabilities:            add:            - NET_BIND_SERVICE            drop:            - all          readOnlyRootFilesystem: true        livenessProbe:          httpGet:            path: /health            port: 8080            scheme: HTTP          initialDelaySeconds: 60          timeoutSeconds: 5          successThreshold: 1          failureThreshold: 5        readinessProbe:          httpGet:            path: /ready            port: 8181            scheme: HTTP      dnsPolicy: Default      volumes:        - name: config-volume          configMap:            name: coredns            items:            - key: Corefile              path: Corefile---apiVersion: v1kind: Servicemetadata:  name: kube-dns  namespace: kube-system  annotations:    prometheus.io/port: "9153"    prometheus.io/scrape: "true"  labels:    k8s-app: kube-dns    kubernetes.io/cluster-service: "true"    kubernetes.io/name: "CoreDNS"spec:  selector:    k8s-app: kube-dns  clusterIP: 10.0.0.10  ports:  - name: dns    port: 53    protocol: UDP  - name: dns-tcp    port: 53    protocol: TCP  - name: metrics    port: 9153    protocol: TCPEOF# To make coredns Only in master Run the node and modify the above file    spec:      priorityClassName: system-cluster-critical      serviceAccountName: coredns      tolerations:        - key: "CriticalAddonsOnly"          operator: "Exists"        - key: "node-role.kubernetes.io/master"          operator: "Exists"      nodeName: k8s-master-1# View podnamespace name ready status restarts age IP node normalized node ready gatecube system calico Kube controllers-855445d444-6glmm 1 / 1 running 0 19m 10.70.2.1 k8s-node-1 < none > < none > Kube system calico node-6pkz6                     1/1     Running   0          19m     192.168.0.10   k8s-master-1   <none>           <none>kube-system   calico-node-8nz7s                          1/1     Running   0          19m     192.168.0.12   k8s-node-2     <none>           <none>kube-system   calico-node-z7pwc                          1/1     Running   0          19m      192.168.0.11   k8s-node-1     <none>           <none>kube-system   coredns-6f4c9cb7c5-hj9bj                   1/1     Running   0          5m48s   10.70.2.66     k8s-master-1   <none>           <none>

Deploy metric server

# establish metrics-server Configuration file, here I set it only in k8s-master-1 The node runs, and readers can modify it according to their own ideas cat > metrics-server.yaml <<EOFapiVersion: v1kind: ServiceAccountmetadata:  labels:    k8s-app: metrics-server  name: metrics-server  namespace: kube-system---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata:  labels:    k8s-app: metrics-server    rbac.authorization.k8s.io/aggregate-to-admin: "true"    rbac.authorization.k8s.io/aggregate-to-edit: "true"    rbac.authorization.k8s.io/aggregate-to-view: "true"  name: system:aggregated-metrics-readerrules:- apiGroups:  - metrics.k8s.io  resources:  - pods  - nodes  verbs:  - get  - list  - watch---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata:  labels:    k8s-app: metrics-server  name: system:metrics-serverrules:- apiGroups:  - ""  resources:  - pods  - nodes  - nodes/stats  - namespaces  - configmaps  verbs:  - get  - list  - watch---apiVersion: rbac.authorization.k8s.io/v1kind: RoleBindingmetadata:  labels:    k8s-app: metrics-server  name: metrics-server-auth-reader  namespace: kube-systemroleRef:  apiGroup: rbac.authorization.k8s.io  kind: Role  name: extension-apiserver-authentication-readersubjects:- kind: ServiceAccount  name: metrics-server  namespace: kube-system---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata:  labels:    k8s-app: metrics-server  name: metrics-server:system:auth-delegatorroleRef:  apiGroup: rbac.authorization.k8s.io  kind: ClusterRole  name: system:auth-delegatorsubjects:- kind: ServiceAccount  name: metrics-server  namespace: kube-system---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata:  labels:    k8s-app: metrics-server  name: system:metrics-serverroleRef:  apiGroup: rbac.authorization.k8s.io  kind: ClusterRole  name: system:metrics-serversubjects:- kind: ServiceAccount  name: metrics-server  namespace: kube-system---apiVersion: v1kind: Servicemetadata:  labels:    k8s-app: metrics-server  name: metrics-server  namespace: kube-systemspec:  ports:  - name: https    port: 443    protocol: TCP    targetPort: https  selector:    k8s-app: metrics-server---apiVersion: apps/v1kind: Deploymentmetadata:  labels:    k8s-app: metrics-server  name: metrics-server  namespace: kube-systemspec:  selector:    matchLabels:      k8s-app: metrics-server  strategy:    rollingUpdate:      maxUnavailable: 0  template:    metadata:      labels:        k8s-app: metrics-server    spec:      nodeName: k8s-master-1      tolerations:      - key: "node-role.kubernetes.io/master"        operator: "Exists"      priorityClassName: system-cluster-critical      serviceAccountName: metrics-server      containers:      - args:        - --cert-dir=/tmp        - --secure-port=4443        - --metric-resolution=30s        - --kubelet-insecure-tls        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname        - --requestheader-client-ca-file=/etc/kubernetes/ssl/front-proxy-ca.pem        - --requestheader-username-headers=X-Remote-User        - --requestheader-group-headers=X-Remote-Group        - --requestheader-extra-headers-prefix=X-Remote-Extra-        image: registry.aliyuncs.com/google_containers/metrics-server:v0.4.1        imagePullPolicy: IfNotPresent        livenessProbe:          failureThreshold: 3          httpGet:            path: /livez            port: https            scheme: HTTPS          periodSeconds: 10        name: metrics-server        ports:        - containerPort: 4443          name: https          protocol: TCP        readinessProbe:          failureThreshold: 3          httpGet:            path: /readyz            port: https            scheme: HTTPS          periodSeconds: 10        securityContext:          readOnlyRootFilesystem: true          runAsNonRoot: true          runAsUser: 1000        volumeMounts:        - mountPath: /tmp          name: tmp-dir        - name: ca-ssl          mountPath: /etc/kubernetes/ssl      volumes:      - emptyDir: {}        name: tmp-dir      - name: ca-ssl        hostPath:          path: /etc/kubernetes/ssl---apiVersion: apiregistration.k8s.io/v1kind: APIServicemetadata:  labels:    k8s-app: metrics-server  name: v1beta1.metrics.k8s.iospec:  group: metrics.k8s.io  groupPriorityMinimum: 100  insecureSkipTLSVerify: true  service:    name: metrics-server    namespace: kube-system  version: v1beta1  versionPriority: 100EOF# obtain node information[root@k8s-master-1 ~]# kubectl top nodesNAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   k8s-master-1   146m         7%     1352Mi          35%       k8s-node-1     84m          4%     726Mi           25%       k8s-node-2     78m          3%     651Mi           22% # see pod function[root@k8s-master-1 ~]# kubectl get pods -A -o wideNAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE    IP             NODE           NOMINATED NODE   READINESS GATESkube-system   calico-kube-controllers-855445d444-6glmm   1/1     Running   0          88m    10.70.2.1      k8s-node-1     <none>           <none>kube-system   calico-node-6pkz6                          1/1     Running   0          88m    192.168.0.10   k8s-master-1   <none>           <none>kube-system   calico-node-8nz7s                          1/1     Running   0          88m    192.168.0.12   k8s-node-2     <none>           <none>kube-system   calico-node-z7pwc                          1/1     Running   0          88m    192.168.0.11   k8s-node-1     <none>           <none>kube-system   coredns-6f4c9cb7c5-hj9bj                   1/1     Running   0          74m    10.70.2.66     k8s-master-1   <none>           <none>kube-system   metrics-server-68bdbcc6b-gk6cq             1/1     Running   0          6m5s   10.70.2.68     k8s-master-1   <none>           <none>

Test cluster network

Note: busybox is better to use 1.28, and the latest version has a BUG

Create test pod

cat << EOF | kubectl apply -f -[root@k8s-master-1 ssl]# cat test-network.yaml apiVersion: v1kind: Podmetadata:  name: busybox-1  namespace: defaultspec:  nodeSelector:    node-role.kubernetes.io/master: ""  tolerations:  - key: node-role.kubernetes.io/master    operator: Exists  containers:  - name: busybox    image: busybox:1.28    imagePullPolicy: IfNotPresent    command:    - sleep    - "86400"  restartPolicy: OnFailure---apiVersion: v1kind: Podmetadata:  name: busybox-2  namespace: defaultspec:  nodeSelector:    node-role.kubernetes.io/node: ""  containers:  - name: busybox    image: busybox:1.28    imagePullPolicy: IfNotPresent    command:    - sleep    - "86400"  restartPolicy: OnFailureEOF# View existing svc[root@k8s-master-1 ssl]# kubectl get svc -ANAMESPACE     NAME             TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGEdefault       kubernetes       ClusterIP   10.0.0.1     <none>        443/TCP                  45hkube-system   kube-dns         ClusterIP   10.0.0.10    <none>        53/UDP,53/TCP,9153/TCP   9hkube-system   metrics-server   ClusterIP   10.0.0.168   <none>        443/TCP                  8h# View current pod Operation[root@k8s-master-1 ssl]# kubectl get pods -A -o wideNAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE   IP             NODE           NOMINATED NODE   READINESS GATESdefault       busybox-1                                  1/1     Running   0          18s   10.70.2.73     k8s-master-1   <none>           <none>default       busybox-2                                  1/1     Running   0          18s   10.70.2.130    k8s-node-2     <none>           <none>kube-system   calico-kube-controllers-855445d444-6glmm   1/1     Running   1          10h   10.70.2.2      k8s-node-1     <none>           <none>kube-system   calico-node-6pkz6                          1/1     Running   1          10h   192.168.0.10   k8s-master-1   <none>           <none>kube-system   calico-node-8nz7s                          1/1     Running   1          10h   192.168.0.12   k8s-node-2     <none>           <none>kube-system   calico-node-z7pwc                          1/1     Running   1          10h   192.168.0.11   k8s-node-1     <none>           <none>kube-system   coredns-6f4c9cb7c5-hj9bj                   1/1     Running   1          10h   10.70.2.69     k8s-master-1   <none>           <none>kube-system   metrics-server-68bdbcc6b-gk6cq             1/1     Running   1          9h    10.70.2.70     k8s-master-1   <none>           <none>

Test pod parsing service

# Test parsing the same namespace Lower service[root@k8s-master-1 ssl]# kubectl exec busybox-1 -- nslookup kubernetesServer:    10.0.0.10Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.localName:      kubernetesAddress 1: 10.0.0.1 kubernetes.default.svc.cluster.local# Span namespace analysis service[root@k8s-master-1 ssl]# kubectl exec busybox-1 -- nslookup kube-dns.kube-systemServer:    10.0.0.10Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.localName:      kube-dns.kube-systemAddress 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local

Test node access kubernetes svc

# Test each node[root@k8s-master-1 ~]# telnet 10.0.0.1 443Trying 10.0.0.1...Connected to 10.0.0.1.Escape character is '^]'.^CConnection closed by foreign host.[root@k8s-master-1 ~]# telnet 10.0.0.10 53Trying 10.0.0.10...Connected to 10.0.0.10.Escape character is '^]'.

Test communication between pod s

[root@k8s-master-1 ssl]# kubectl exec busybox-1 -- ping 10.70.2.130PING 10.70.2.130 (10.70.2.130): 56 data bytes64 bytes from 10.70.2.130: seq=0 ttl=62 time=0.603 ms64 bytes from 10.70.2.130: seq=1 ttl=62 time=0.451 ms

Note:

Communicate with the namespace
Be able to communicate across namespace s
Cross machine communication

Tags: Linux Docker Kubernetes

Posted on Wed, 06 Oct 2021 08:43:55 -0400 by fisicx

Programmer Help

k8s-v1.20.10 1master&2node binary deployment guidance document

k8s-v1.20.10 1master&2node

Experimental environment

Host network information and component information

Host certificate information

Host initialization

Configure host name

Configure hosts file

Secret free login

Turn off firewall

Close swap partition

Configure yum source

Configure synchronization time

Upgrade kernel

Modify kernel parameters

Install docker CE

Install cfssl

CA initialization

etcd-ca

kube-apiserver-ca

front-proxy-ca

Deploy etcd

Create etcd certificate

Create etcd profile

Deploy apiserver

Upload k8s component

Create a token.csv file

Create apiserver certificate

Create front proxy CA certificate

Create service public / private key

Create apiserver configuration file

Deploy kubectl

Create kubctl certificate

Create kubeconfig configuration file for kubectl

Deploy Kube Controller Manager

Create Kube controller manager certificate

Create kubeconfig for Kube Controller Manager

Create Kube controller manager configuration file

Deploy Kube scheduler

Create Kube scheduler certificate

Create kubeconfig for Kube scheduler

Create Kube scheduler configuration file

Deploy kubelet

Create kubeconfig for kubelet

Create kublet's profile

Create kubelet startup file

Create RBAC rules to automatically approve CSR

Node deployment Kubelet

Deploy Kube proxy

Create Kube proxy certificate

Create kubeconfig for Kube proxy

Create Kube proxy profile

Create Kube proxy startup file

Node deployment Kube proxy

Add cluster role

Deploy calico

Deploy coredns

Deploy metric server

Test cluster network

Create test pod

Test pod parsing service

Test node access kubernetes svc

Test communication between pod s

Hot Tags