Pod details - lifecycle - container detection

Container detection is used to detect whether the application instances in the container work normally. It is a traditional mechanism to ensure service availability. If the status of the instance does not meet the expectation after detection, kubernetes will "remove" the problem instance and do not undertake business traffic. Kubernetes provides two probes to realize container detection, namely:

  • liveness probes: Live probes are used to detect whether the application instance is currently in normal operation. If not, k8s the container will be restarted

  • readiness probes: readiness probes are used to detect whether the application instance can receive requests. If not, k8s it will not forward traffic

livenessProbe decides whether to restart the container, and readinessProbe decides whether to forward the request to the container.

The above two probes currently support three detection modes:

  • Exec command: execute the command once in the container. If the exit code of the command is 0, the program is considered normal, otherwise it is not normal

......
  livenessProbe:
    exec:
      command:
      - cat
      - /tmp/healthy
......

TCPSocket: will try to access the port of a user container. If this connection can be established, the program is considered normal, otherwise it is not normal

......      
  livenessProbe:
    tcpSocket:
      port: 8080
......

HTTPGet: call the URL of the Web application in the container. If the returned status code is between 200 and 399, the program is considered normal, otherwise it is not normal

......
  livenessProbe:
    httpGet:
      path: / #URI address
      port: 80 #Port number
      host: 127.0.0.1 #Host address
      scheme: HTTP #Supported protocols, http or https
......

Let's take liveness probes as an example to demonstrate:

Method 1: Exec

Create pod-liveness-exec.yaml

apiVersion: v1
kind: Pod
metadata:
  name: pod-liveness-exec
  namespace: dev
spec:
  containers:
  - name: nginx
    image: nginx:1.17.1
    ports: 
    - name: nginx-port
      containerPort: 80
    livenessProbe:
      exec:
        command: ["/bin/cat","/tmp/hello.txt"] # Execute a command to view files

Create a pod and observe the effect

# Create Pod
[root@k8s-master01 ~]# kubectl create -f pod-liveness-exec.yaml
pod/pod-liveness-exec created

# View Pod details
[root@k8s-master01 ~]# kubectl describe pods pod-liveness-exec -n dev
......
  Normal   Created    20s (x2 over 50s)  kubelet, node1     Created container nginx
  Normal   Started    20s (x2 over 50s)  kubelet, node1     Started container nginx
  Normal   Killing    20s                kubelet, node1     Container nginx failed liveness probe, will be restarted
  Warning  Unhealthy  0s (x5 over 40s)   kubelet, node1     Liveness probe failed: cat: can't open '/tmp/hello11.txt': No such file or directory
  
# Observing the above information, you will find that the health check is carried out after the nginx container is started
# After the check fails, the container is kill ed, and then restart is attempted (this is the function of the restart strategy, which will be explained later)
# After a while, observe the pod information, and you can see that the RESTARTS is no longer 0, but has been growing
[root@k8s-master01 ~]# kubectl get pods pod-liveness-exec -n dev
NAME                READY   STATUS             RESTARTS   AGE
pod-liveness-exec   0/1     CrashLoopBackOff   2          3m19s

# Of course, next, you can modify it to an existing file, such as / tmp/hello.txt. Try again, and the result will be normal

Mode 2: TCPSocket

Create pod-liveness-tcpsocket.yaml

apiVersion: v1
kind: Pod
metadata:
  name: pod-liveness-tcpsocket
  namespace: dev
spec:
  containers:
  - name: nginx
    image: nginx:1.17.1
    ports: 
    - name: nginx-port
      containerPort: 80
    livenessProbe:
      tcpSocket:
        port: 8080 # Trying to access port 8080

Create a pod and observe the effect

# Create Pod
[root@k8s-master01 ~]# kubectl create -f pod-liveness-tcpsocket.yaml
pod/pod-liveness-tcpsocket created

# View Pod details
[root@k8s-master01 ~]# kubectl describe pods pod-liveness-tcpsocket -n dev
......
  Normal   Scheduled  31s                            default-scheduler  Successfully assigned dev/pod-liveness-tcpsocket to node2
  Normal   Pulled     <invalid>                      kubelet, node2     Container image "nginx:1.17.1" already present on machine
  Normal   Created    <invalid>                      kubelet, node2     Created container nginx
  Normal   Started    <invalid>                      kubelet, node2     Started container nginx
  Warning  Unhealthy  <invalid> (x2 over <invalid>)  kubelet, node2     Liveness probe failed: dial tcp 10.244.2.44:8080: connect: connection refused
  
# Observing the above information, I found that I tried to access port 8080, but failed
# After a while, observe the pod information, and you can see that the RESTARTS is no longer 0, but has been growing
[root@k8s-master01 ~]# kubectl get pods pod-liveness-tcpsocket  -n dev
NAME                     READY   STATUS             RESTARTS   AGE
pod-liveness-tcpsocket   0/1     CrashLoopBackOff   2          3m19s

# Of course, next, you can change it to an accessible port, such as 80. Try again, and the result will be normal

Method 3: HTTPGet

Create pod-liveness-httpget.yaml

apiVersion: v1
kind: Pod
metadata:
  name: pod-liveness-httpget
  namespace: dev
spec:
  containers:
  - name: nginx
    image: nginx:1.17.1
    ports:
    - name: nginx-port
      containerPort: 80
    livenessProbe:
      httpGet:  # It's actually a visit http://127.0.0.1:80/hello  
        scheme: HTTP #Supported protocols, http or https
        port: 80 #Port number
        path: /hello #URI address

Create a pod and observe the effect

# Create Pod
[root@k8s-master01 ~]# kubectl create -f pod-liveness-httpget.yaml
pod/pod-liveness-httpget created

# View Pod details
[root@k8s-master01 ~]# kubectl describe pod pod-liveness-httpget -n dev
.......
  Normal   Pulled     6s (x3 over 64s)  kubelet, node1     Container image "nginx:1.17.1" already present on machine
  Normal   Created    6s (x3 over 64s)  kubelet, node1     Created container nginx
  Normal   Started    6s (x3 over 63s)  kubelet, node1     Started container nginx
  Warning  Unhealthy  6s (x6 over 56s)  kubelet, node1     Liveness probe failed: HTTP probe failed with statuscode: 404
  Normal   Killing    6s (x2 over 36s)  kubelet, node1     Container nginx failed liveness probe, will be restarted
  
# Observe the above information and try to access the path, but it is not found. A 404 error occurs
# After a while, observe the pod information, and you can see that the RESTARTS is no longer 0, but has been growing
[root@k8s-master01 ~]# kubectl get pod pod-liveness-httpget -n dev
NAME                   READY   STATUS    RESTARTS   AGE
pod-liveness-httpget   1/1     Running   5          3m17s

# Of course, next, you can modify it to an accessible path, such as /. Try again, and the result will be normal

So far, the liveness Probe has been used to demonstrate three detection methods. However, looking at the child properties of liveness Probe, you will find that there are other configurations in addition to these three methods, which are explained here:

[root@k8s-master01 ~]# kubectl explain pod.spec.containers.livenessProbe
FIELDS:
   exec <Object>  
   tcpSocket    <Object>
   httpGet      <Object>
   initialDelaySeconds  <integer>  # How many seconds do you wait for the first probe after the container starts
   timeoutSeconds       <integer>  # Probe timeout. Default 1 second, minimum 1 second
   periodSeconds        <integer>  # The frequency at which the probe is performed. The default is 10 seconds and the minimum is 1 second
   failureThreshold     <integer>  # How many times does a continuous probe fail before it is considered a failure. The default is 3. The minimum value is 1
   successThreshold     <integer>  # How many times are successive detections considered successful. The default is 1

Here are two slightly configured to demonstrate the following effects:

[root@k8s-master01 ~]# more pod-liveness-httpget.yaml
apiVersion: v1
kind: Pod
metadata:
  name: pod-liveness-httpget
  namespace: dev
spec:
  containers:
  - name: nginx
    image: nginx:1.17.1
    ports:
    - name: nginx-port
      containerPort: 80
    livenessProbe:
      httpGet:
        scheme: HTTP
        port: 80 
        path: /
      initialDelaySeconds: 30 # Detection starts 30 s after the container is started
      timeoutSeconds: 5 # The detection timeout is 5s

Tags: Kubernetes

Posted on Sat, 20 Nov 2021 09:09:10 -0500 by packland