Kubernetes StatefulSet for stateful service data persistence (automatic creation of PVC)

1. Kubernetes Stateless Service VS Stateful Service

1) Kubernetes Stateless Service

Kubernetes stateless service characteristics:
1) means that instances running the service do not store data that needs to be persisted locally and that multiple instances respond to the same request with identical results;
2) Multiple instances can share the same persisted data.For example, nginx instance, tomcat instance, etc.
3) The related Kubernetes resources are ReplicaSet, ReplicationController, Deployment, etc. Because they are stateless services, the PD names created by these controllers are random.And when scaling, it is not explicit to scale a Pod, but random, because all instances get the same return value, so any Pod can be scaled;

2) Kubernetes stateful service

Kubernetes has stateful service characteristics:
1) Stateful services can be referred to as services that require data storage capabilities, or services, queues, etc. of a multithreaded type.(e.g. mysql database, kafka, zookeeper, etc.);
2) Each instance needs its own independent persistent storage and is defined in Kubernetes by declaring templates.Persistent volume declaration templates are created prior to pod creation and are bound to pods, where templates can define multiple;
3) The related Kubernetes resources are StatefulSet.Since there are stateful services, each Pod has a specific name and network identity.For example, a Pod name is a StatefulSet name + an ordered number (0, 1, 2...);
4) When scaling, you can know which Pod will be scaled, starting with the largest number.And StatefulSet is not allowed to shrink if existing instances are unhealthy;

3) Differences between stateless and stateful services

The main manifestations are as follows:
1) Number of instances: A stateless service can have one or more instances, thus supporting two service capacity regulation modes; a stateful service function has one instance that does not allow multiple instances to be created, and therefore does not support service capacity regulation;
2) Storage volumes: Stateless services can have or cannot back up data in storage volumes, even if they do; Stateless services must have storage volumes, and when creating services, the size of disk space allocated by the storage volumes must be specified;
3) Data Storage: All data in the running process of stateless service (except log and monitoring data) exists in the file system in the container instance, and if the instance stops or deletes, the data will be lost and cannot be retrieved. For stateful service, the contents of files in the directory where the storage volume has been mounted can be backed up at any time, and the backed up dataIt can be downloaded or used to restore new services.However, data in directories that do not have mounted volumes can still not be backed up and saved. If instances are stopped or deleted, the contents of files in these non-mounted volumes will also be lost.

4) Overview of StatefulSet

StatefulSet is a load management controller API provided by Kubernetes to manage stateful applications.On the basis of Pods management, ensure the order and consistency of Pods.StatefulSet, like Deployment, uses a container's Spec to create a Pod, unlike StatefulSet, which creates a Pod that maintains a persistent marker (such as a Pod Name) throughout its life cycle.

5) Features of StatefulSet

1) Stable persistent storage, i.e. access to the same persistent data after Pod rescheduling, based on PVC;
2) Stable network flags, i.e. PodName and HostName remain unchanged after Pod is rescheduled, based on Headless Service (Service without Cluster IP);
3) Ordered deployment, ordered expansion, that is, Pod is ordered, deployed or expanded in the order defined (that is, from 0 to N-1, all previous Pods must be Running and Ready before the next Pod runs), based on init containers;
4) Orderly contraction, orderly deletion (that is, from N-1 to 0);

2. Auto-create PVC using StatefulSet

1) Build NFS Shared Storage

For convenience, deploy NFS storage directly on the master node!

[root@master ~]# yum -y install nfs-utils rpcbind
[root@master ~]# vim /etc/exports
/nfsdata *(rw,sync,no_root_squash)
[root@master ~]# mkdir /nfsdata
[root@master ~]# systemctl start nfs-server
[root@master ~]# systemctl start rpcbind
[root@master ~]# showmount -e
Export list for master:
/nfsdata *

2) Create rbac authorization

[root@master ~]# vim rbac-rolebind.yaml
apiVersion: v1                            #Create a service account for authentication
kind: ServiceAccount
metadata:
  name: nfs-provisioner
---
apiVersion: rbac.authorization.k8s.io/v1        #Create Cluster Rules
kind: ClusterRole
metadata:
  name: nfs-provisioner-runner
rules:
   -  apiGroups: [""]
      resources: ["persistentvolumes"]
      verbs: ["get", "list", "watch", "create", "delete"]
   -  apiGroups: [""]
      resources: ["persistentvolumeclaims"]
      verbs: ["get", "list", "watch", "update"]
   -  apiGroups: ["storage.k8s.io"]
      resources: ["storageclasses"]
      verbs: ["get", "list", "watch"]
   -  apiGroups: [""]
      resources: ["events"]
      verbs: ["watch", "create", "update", "patch"]
   -  apiGroups: [""]
      resources: ["services", "endpoints"]
      verbs: ["get","create","list", "watch","update"]
   -  apiGroups: ["extensions"]
      resources: ["podsecuritypolicies"]
      resourceNames: ["nfs-provisioner"]
      verbs: ["use"]
---
kind: ClusterRoleBinding                #Binding service authenticated users to cluster rules
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-provisioner
    namespace: default                    #Required paragraph, otherwise error will be prompted
roleRef:
  kind: ClusterRole
  name: nfs-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
[root@master ~]# kubectl apply -f rbac-rolebind.yaml    

3) Create nfs-deployment. Resources

[root@master ~]# vim nfs-deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nfs-client-provisioner
spec:
  replicas: 1                              #Number of copies specified is 1
  strategy:
    type: Recreate                      #Specify policy type as reset
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccount: nfs-provisioner            #Specify the authenticated user account created in the rbac yanl file
      containers:
        - name: nfs-client-provisioner
          image: registry.cn-hangzhou.aliyuncs.com/open-ali/nfs-client-provisioner     #Mirror used 
          volumeMounts:
            - name: nfs-client-root
              mountPath:  /persistentvolumes             #Specify the directory mounted in the container
          env:
            - name: PROVISIONER_NAME           #Variables inside containers are used to specify the name of the storage being provided
              value: lzj
            - name: NFS_SERVER                      #Variables inside the container are used to specify the IP address of the nfs service
              value: 192.168.1.1
            - name: NFS_PATH                       #Variables inside the container specify the directory corresponding to the nfs server
              value: /nfsdata
      volumes:                                                #Specify the path and IP of nfs mounted in the container
        - name: nfs-client-root
          nfs:
            server: 192.168.1.1
            path: /nfsdata
[root@master ~]# kubectl apply -f nfs-deployment.yaml
[root@master ~]# kubectl get pod 
NAME                                      READY   STATUS    RESTARTS   AGE
nfs-client-provisioner-66df958f9c-mbvhv   1/1     Running   0          2m34s

4) Create SC (Storage Class)

[root@master ~]# vim sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: stateful-nfs
  namespace: xiaojiang-test
provisioner: lzj                  #This corresponds to the value of PROVISIONER_NAME in the env environment variable of nfs-client-provisioner.
reclaimPolicy: Retain               #Specify Recycle Policy as Retain (Manual Release)
[root@master ~]# kubectl apply -f sc.yaml 
[root@master ~]# kubectl get StorageClass
NAME           PROVISIONER   AGE
stateful-nfs   lzj           17s

5) Create Pod

[root@master ~]# vim statefulset.yaml 
apiVersion: v1
kind: Service
metadata:
  name: headless-svc                    #Headless service can be from name
  labels:
    app: headless-svc
spec:
  ports:
  - port: 80
    name: myweb
  selector:
    app: headless-pod
  clusterIP: None                        #No cluster IP address is assigned, so no load balancing capability is available
---
apiVersion: apps/v1
kind: StatefulSet                          #Define applications running in pod
metadata:
  name: statefulset-test
spec:
  serviceName: headless-svc
  replicas: 3
  selector:
    matchLabels:
      app: headless-pod
  template:
    metadata:
      labels:
        app: headless-pod
    spec:
      containers:
      - image: httpd
        name: myhttpd
        ports:
        - containerPort: 80
          name: httpd
        volumeMounts:
        - mountPath: /usr/local/apache2/htdocs
          name: test
  volumeClaimTemplates:                       #Define the template used to create PVC
  - metadata:
      name: test
      annotations:  #This is the specified storageclass
        volume.beta.kubernetes.io/storage-class: stateful-nfs
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 100Mi
[root@master ~]# kubectl apply -f statefulset.yaml 
[root@master ~]# kubectl get pod
NAME                                      READY   STATUS    RESTARTS   AGE
nfs-client-provisioner-66df958f9c-mbvhv   1/1     Running   0          10m
statefulset-test-0                        1/1     Running   0          29s
statefulset-test-1                        1/1     Running   0          18s
statefulset-test-2                        1/1     Running   0          11s
[root@master ~]# kubectl get pv
[root@master ~]# kubectl get pvc              #PV and PVC have been generated
[root@master ~]# ls /nfsdata/
default-test-statefulset-test-0-pvc-54d0b06c-698e-4f1a-8327-255b10978cbe
default-test-statefulset-test-1-pvc-1b499d49-a787-4f2b-b238-404b05f75fd7
default-test-statefulset-test-2-pvc-7766f8da-6f3b-4c1f-9eb8-dfadda1e656f
[root@master ~]# echo "hello world" > /nfsdata/default-test-statefulset-test-0-pvc-54d0b06c-698e-4f1a-8327-255b10978cbe/index.html
[root@master ~]# kubectl get pod -o wide | grep test-0
statefulset-test-0                        1/1     Running   0          4m53s   10.244.2.4   node02   <none>           <none>
[root@master ~]# curl 10.244.2.4
hello world
[root@master ~]# curl -I 10.244.2.4
HTTP/1.1 200 OK
Date: Wed, 12 Feb 2020 09:52:04 GMT
Server: Apache/2.4.41 (Unix)
Last-Modified: Wed, 12 Feb 2020 09:45:37 GMT
ETag: "c-59e5dd5ac0a63"
Accept-Ranges: bytes
Content-Length: 12
Content-Type: text/html
#You can see that the service that now provides web pages is Apache

6) Update and expand pod

[root@master ~]# vim statefulset.yaml 
apiVersion: v1
kind: Service
metadata:
  name: headless-svc
  labels:
    app: headless-svc
spec:
  ports:
  - port: 80
    name: myweb
  selector:
    app: headless-pod
  clusterIP: None
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: statefulset-test
spec:
  updateStrategy:
    rollingUpdate:
      partition: 2                           #Specify the number of parallel upgrades
  serviceName: headless-svc
  replicas: 10
  selector:
    matchLabels:
      app: headless-pod
  template:
    metadata:
      labels:
        app: headless-pod
    spec:
      containers:
      - image: nginx                       #Replace the mirror used for expansion
        name: myhttpd
        ports:
        - containerPort: 80
          name: httpd
        volumeMounts:
        - mountPath: /usr/share/nginx/html/                 #Replace the home directory in the container
          name: test
  volumeClaimTemplates:
  - metadata:
      name: test
      annotations:  #This is the specified storageclass
        volume.beta.kubernetes.io/storage-class: stateful-nfs
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 100Mi
[root@master ~]# kubectl apply -f statefulset.yaml 
[root@master ~]# kubectl get pod -o wide
NAME                                      READY   STATUS    RESTARTS   AGE     IP           NODE     NOMINATED NODE   READINESS GATES
nfs-client-provisioner-66df958f9c-mbvhv   1/1     Running   0          35m     10.244.2.2   node02   <none>           <none>
statefulset-test-0                        1/1     Running   0          21m     10.244.2.4   node02   <none>           <none>
statefulset-test-1                        1/1     Running   0          20m     10.244.1.4   node01   <none>           <none>
statefulset-test-2                        1/1     Running   0          3m52s   10.244.1.9   node01   <none>           <none>
statefulset-test-3                        1/1     Running   0          4m54s   10.244.2.5   node02   <none>           <none>
statefulset-test-4                        1/1     Running   0          4m43s   10.244.1.6   node01   <none>           <none>
statefulset-test-5                        1/1     Running   0          4m31s   10.244.2.6   node02   <none>           <none>
statefulset-test-6                        1/1     Running   0          4m25s   10.244.1.7   node01   <none>           <none>
statefulset-test-7                        1/1     Running   0          4m19s   10.244.2.7   node02   <none>           <none>
statefulset-test-8                        1/1     Running   0          4m12s   10.244.1.8   node01   <none>           <none>
statefulset-test-9                        1/1     Running   0          4m3s    10.244.2.8   node02   <none>           <none>
[root@master ~]# ls /nfsdata/ | wc -l
10
[root@master ~]# curl -I 10.244.2.4
HTTP/1.1 200 OK
Date: Wed, 12 Feb 2020 10:05:34 GMT
Server: Apache/2.4.41 (Unix)
Last-Modified: Wed, 12 Feb 2020 09:45:37 GMT
ETag: "c-59e5dd5ac0a63"
Accept-Ranges: bytes
Content-Length: 12
Content-Type: text/html
[root@master ~]# curl -I 10.244.2.8
HTTP/1.1 403 Forbidden
Server: nginx/1.17.8
Date: Wed, 12 Feb 2020 10:05:41 GMT
Content-Type: text/html
Content-Length: 153
Connection: keep-alive

It can be seen that the pod is not changed during the expansion operation, which is the characteristic of StatefulSet!

To the end of this article, thank you for reading -----------

Tags: Kubernetes vim Nginx curl

Posted on Wed, 12 Feb 2020 12:35:39 -0500 by Tjeuten