Master container probes for SpringBoot-2.3: Actual Text

Welcome to my GitHub

https://github.com/zq2599/blog_demos

  • Content: A summary of the original articles, together with supporting source code, including Java, Docker, K8S, DevOPS, etc.
    After a lot of knowledge has been accumulated, we have come to the real world chapter. Dear readers, please put your equipment in place and experience the latest technology that SpringBoot officially brings to us.

About the SpringBoot-2.3 Containerization Technology Series

  • The SpringBoot-2.3 Containerized Technology series is designed to learn and practice the latest containerized technology brought about by version 2.3, so that our Java applications can adapt more to the containerized environment, keep up with the mainstream and remain competitive in the cloud computing era.
  • The full series of articles is divided into two parts, the theme and the supplementary part, which are as follows:
  1. Experience SpringBoot(2.3) Apps for Docker Mirroring (Official Program)
  2. Detailed SpringBoot(2.3) Application for Docker Mirroring (Official Plan)
  3. Mastering Container Probes for SpringBoot-2.3: Foundation
  4. Mastering Container Probes for SpringBoot-2.3: An in-depth article
  5. Mastering Container Probes for SpringBoot-2.3: A Practical Text
  • The supplementary section is a summary of some references and memos, as follows:
  1. Why do SpringBoot-2.3 mirroring schemes require multiple layer s?
  2. Setting up a non-root account does not require sudo to execute the docker command directly
  3. "Quick Deployment of SpringBoot Applications to K8S in Development Phase"

Summary of knowledge points of SpringBoot-2.3 container probe

After the previous knowledge buildup, we know the new probe specifications and scenarios for SpringBoot-2.3. Here is a brief review:

  1. kubernetes requires the business container to provide an address called livenessProbe, which is regularly accessed by kubernetes. If the return code of the address is not between 200 and 400, kubernetes considers the container unhealthy and will kill the container to rebuild a new container, which is the survival probe.
  2. kubernetes requires the business container to provide an address called readinessProbe, which is accessed regularly by kubernetes. If the return code of the address is not between 200 and 400, kubernetes does not think the container can provide services to the outside world and will not schedule requests to the container. This address is the ready probe.
  3. SpringBoot's 2.3.0.RELEASE released two new actuator addresses, /actuator/health/liveness and/actuator/health/readiness, which serve as survival probes and/actuator/health/readiness, which return values from two new actuators: Liveness State and Readiness State;
  4. SpringBoot applications determine whether they are running in a container environment based on the presence or absence of a particular environment variable. If so, two addresses, /actuator/health/liveness and/actuator/health/readiness, have return codes. Specific values correspond to the state of the application, for example, /actuator/health/readiness returns 503 during application startup and 200 after successful startup.
  5. Business applications can read Liveness State and Readiness State through the Spring system event mechanism, or subscribe to change events for both actuator s.
  6. Business applications can modify Liveness State and Readiness through Spring system event mechanismsState, at this point, the return values of /actuator/health/liveness and/actuator/health/readiness will change, affecting the behavior of kubernetes with respect to this container (referring to the first and second points), such as the livenessProbe return code becoming 503, causing kubernetes to think the container is unhealthy and kill the container;

When the summary is complete, we will start to practice coding and operation to verify the above theory.

Actual Environmental Information

This battle has two environments: the development and operation environment, in which the development environment information is as follows:

  1. Operating System: Ubuntu 20.04 LTS Desktop Edition
  2. CPU: 2.30GHz x 4, memory: 32G, hard disk: 1T NVMe
  3. JDK: 1.8.0_231
  4. MAVEN: 3.6.3
  5. SpringBoot: 2.3.0.RELEASE
  6. Docker: 19.03.10
  7. Development Tools: IDEA 2020.1.1 (Ultimate Edition)

The running environment information is as follows:

  1. Operating System: CentOS Linux release 7.8.2003
  2. Kubernetes: 1.15

It has proved that using Ubuntu Desktop Edition as the development environment is feasible, and the experience is very smooth. IDEA, SubLime, SSH, Chrome, WeChat can be used normally. The following image is my Ubuntu development environment:

Brief introduction of actual combat content

This actual battle includes the following:

  1. Develop SpringBoot applications and deploy them in kubernetes;
  2. Check the associated changes between the application state and the pod state of kubernetes;
  3. Modify Readiness State to see if kubernetes will also schedule requests to pod s;
  4. Modify Liveness State to see if kubernetes kills the pod;

Source Download

  1. A common SpringBoot project was used in this battle. The source code can be downloaded from GitHub, and the address and link information are shown in the table below. https://github.com/zq2599/blog_demos):
Name link Remarks
Project Home https://github.com/zq2599/blog_demos The project's home page on GitHub
git repository address (https) https://github.com/zq2599/blog_demos.git Warehouse address for the project source code, https protocol
git warehouse address (ssh) git@github.com:zq2599/blog_demos.git Warehouse address for the project source code, ssh protocol
  1. There are several folders in this git project. The application of this chapter is under the probedemo folder, as shown in the red box below:

Developing SpringBoot applications

  1. Install lombok plug-in on IDEA:

  1. Create a new SpringBoot project named probedemo on IDEA with version 2.3.0:

  1. Of the projectPom.xmlAs follows, be aware of the spring-boot-starter-actuator and lombok dependencies, and add the layers node to the plug-in spring-boot-maven-plugin:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.3.0.RELEASE</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.bolingcavalry</groupId>
    <artifactId>probedemo</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>probedemo</name>
    <description>Demo project for Spring Boot</description>

    <properties>
        <java.version>1.8</java.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
            <exclusions>
                <exclusion>
                    <groupId>org.junit.vintage</groupId>
                    <artifactId>junit-vintage-engine</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>

        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <version>2.3.0.RELEASE</version>
                <!--The configuration will be jar Medium increase layer Description file, and extraction layer Tools-->
                <configuration>
                    <layers>
                        <enabled>true</enabled>
                    </layers>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>
  1. The application startup class ProbedemoApplication is the most common startup class:
package com.bolingcavalry.probedemo;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class ProbedemoApplication {
    public static void main(String[] args) {
        SpringApplication.run(ProbedemoApplication.class, args);
    }
}
  1. Add a listening class to monitor changes in survival and readiness:
package com.bolingcavalry.probedemo.listener;

import lombok.extern.slf4j.Slf4j;
import org.springframework.boot.availability.AvailabilityChangeEvent;
import org.springframework.boot.availability.AvailabilityState;
import org.springframework.context.event.EventListener;
import org.springframework.stereotype.Component;

/**
 * description: Class listening for system events <br>
 * date: 2020/6/4 12:57 p.m. <br>
 * author: willzhao <br>
 * email: zq2599@gmail.com <br>
 * version: 1.0 <br>
 */
@Component
@Slf4j
public class AvailabilityListener {

    /**
     * Listen for system messages,
     * AvailabilityChangeEvent Messages of type are called back from the method that triggers it
     * @param event
     */
    @EventListener
    public void onStateChange(AvailabilityChangeEvent<? extends AvailabilityState> event) {
        log.info(event.getState().getClass().getSimpleName() + " : " + event.getState());
    }
}
  1. Add a Controller named StateReader for survival and readiness:
package com.bolingcavalry.probedemo.controller;

import org.springframework.boot.availability.ApplicationAvailability;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import javax.annotation.Resource;
import java.util.Date;

@RestController
@RequestMapping("/statereader")
public class StateReader {

    @Resource
    ApplicationAvailability applicationAvailability;

    @RequestMapping(value="/get")
    public String state() {
        return "livenessState : " + applicationAvailability.getLivenessState()
               + "<br>readinessState : " + applicationAvailability.getReadinessState()
               + "<br>" + new Date();
    }
}
  1. Add a Controller named StateWritter to set the state of survival and readiness:
package com.bolingcavalry.probedemo.controller;

import org.springframework.boot.availability.AvailabilityChangeEvent;
import org.springframework.boot.availability.LivenessState;
import org.springframework.boot.availability.ReadinessState;
import org.springframework.context.ApplicationEventPublisher;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import javax.annotation.Resource;
import java.util.Date;

/**
 * description: Modify state controller <br>
 * date: 2020/6/4 1:21 p.m. <br>
 * author: willzhao <br>
 * email: zq2599@gmail.com <br>
 * version: 1.0 <br>
 */
@RestController
@RequestMapping("/staterwriter")
public class StateWritter {

    @Resource
    ApplicationEventPublisher applicationEventPublisher;

    /**
     * Change survival status to BROKEN (causes kubernetes to kill pod)
     * @return
     */
    @RequestMapping(value="/broken")
    public String broken(){
        AvailabilityChangeEvent.publish(applicationEventPublisher, StateWritter.this, LivenessState.BROKEN);
        return "success broken, " + new Date();
    }

    /**
     * Change Survival Status to CORRECT
     * @return
     */
    @RequestMapping(value="/correct")
    public String correct(){
        AvailabilityChangeEvent.publish(applicationEventPublisher, StateWritter.this, LivenessState.CORRECT);
        return "success correct, " + new Date();
    }

    /**
     * Change Ready State to REFUSING_TRAFFIC (causes kubernetes to no longer forward external requests to this pod)
     * @return
     */
    @RequestMapping(value="/refuse")
    public String refuse(){
        AvailabilityChangeEvent.publish(applicationEventPublisher, StateWritter.this, ReadinessState.REFUSING_TRAFFIC);
        return "success refuse, " + new Date();
    }

    /**
     * Change Ready State to ACCEPTING_TRAFFIC (causes kubernetes to forward external requests to this pod)
     * @return
     */
    @RequestMapping(value="/accept")
    public String accept(){
        AvailabilityChangeEvent.publish(applicationEventPublisher, StateWritter.this, ReadinessState.ACCEPTING_TRAFFIC);
        return "success accept, " + new Date();
    }

}
  1. Add a controller named Hello, which returns the IP address of the current pod and will be used in subsequent tests:
package com.bolingcavalry.probedemo.controller;

import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import java.net.Inet4Address;
import java.net.InetAddress;
import java.net.NetworkInterface;
import java.net.SocketException;
import java.util.ArrayList;
import java.util.Date;
import java.util.Enumeration;
import java.util.List;

/**
 * description: hello demo <br>
 * date: 2020/6/4 4:38 p.m. <br>
 * author: willzhao <br>
 * email: zq2599@gmail.com <br>
 * version: 1.0 <br>
 */
@RestController
public class Hello {

    /**
     * Returns the current server IP address, which is the pod address in the k8s environment
     * @return
     * @throws SocketException
     */
    @RequestMapping(value="/hello")
    public String hello() throws SocketException {
        List<Inet4Address> addresses = getLocalIp4AddressFromNetworkInterface();
        if(null==addresses || addresses.isEmpty()) {
            return  "empty ip address, " + new Date();
        }

        return addresses.get(0).toString() + ", " + new Date();
    }

    public static List<Inet4Address> getLocalIp4AddressFromNetworkInterface() throws SocketException {
        List<Inet4Address> addresses = new ArrayList<>(1);
        Enumeration e = NetworkInterface.getNetworkInterfaces();
        if (e == null) {
            return addresses;
        }
        while (e.hasMoreElements()) {
            NetworkInterface n = (NetworkInterface) e.nextElement();
            if (!isValidInterface(n)) {
                continue;
            }
            Enumeration ee = n.getInetAddresses();
            while (ee.hasMoreElements()) {
                InetAddress i = (InetAddress) ee.nextElement();
                if (isValidAddress(i)) {
                    addresses.add((Inet4Address) i);
                }
            }
        }
        return addresses;
    }

    /**
     * Filter the loopback network card, peer-to-peer network card, inactive network card, virtual network card and require the network card name to start with eth or ens
     * @param ni Network card
     * @return true if required, false otherwise
     */
    private static boolean isValidInterface(NetworkInterface ni) throws SocketException {
        return !ni.isLoopback() && !ni.isPointToPoint() && ni.isUp() && !ni.isVirtual()
                && (ni.getName().startsWith("eth") || ni.getName().startsWith("ens"));
    }

    /**
     * Determine if it is IPv4 and filter the loopback address.
     */
    private static boolean isValidAddress(InetAddress address) {
        return address instanceof Inet4Address && address.isSiteLocalAddress() && !address.isLoopbackAddress();
    }
}

That's all the code for the SpringBoot project. Make sure it can be compiled and run.

Making Docker Mirrors

  1. stayPom.xmlThe file Dockerfile is created in your directory as follows:
# Specify the base image, which is an early stage of a phased build
FROM openjdk:8u212-jdk-stretch as builder
# Execution Working Directory
WORKDIR application
# configuration parameter
ARG JAR_FILE=target/*.jar
# Copy the compiled jar file into the mirror space
COPY ${JAR_FILE} application.jar
# From the tool spring-boot-jarmode-layertools Application.jarExtract the split build result
RUN java -Djarmode=layertools -jar application.jar extract

# Formally Build Mirror
FROM openjdk:8u212-jdk-stretch
WORKDIR application
# The previous phase extracted multiple files from jar, where COPY commands are executed and copied into the mirror space, each time COPY is a layer
COPY --from=builder application/dependencies/ ./
COPY --from=builder application/spring-boot-loader/ ./
COPY --from=builder application/snapshot-dependencies/ ./
COPY --from=builder application/application/ ./
ENTRYPOINT ["java", "org.springframework.boot.loader.JarLauncher"]
  1. Compile the build project first and execute the following commands:
mvn clean package -U -DskipTests 
  1. After successful compilation, create a mirror from the Dockerfile file:
sudo docker build -t bolingcavalry/probedemo:0.0.1 .
  1. Mirror created successfully:


The mirror of SpringBoot is ready to be used by the kubernetes environment.

Loading a mirror into the kubernetes environment

At this point, the mirror is saved on the computer in the development environment and can be loaded into the kubernetes environment in three ways:

  1. push to private repositories, which are also acquired from private repositories when used by kubernetes;
  2. push toHub.docker.com, kubernetes is also used fromHub.docker.comGet, I've now pushed this image toHub.docker.com, you can use it directly in kubernetes, just like the official mirrors of nginx, tomcat download;
  3. Execute docker save bolingcavalry/Probedemo:0.0.1>Probedemo.tar, you can save the image as a local file, scp it to the kubernetes server, and execute docker load </root/temp/202006/04/Probedemo.tarCan be loaded into the local docker cache of the kubernetes server;

The advantages and disadvantages of the above three methods are summarized as follows:

  1. First, but you need to build a private warehouse.
  2. Because springboot-2.3 officially optimizes mirror building, the second method takes a lot of time to upload and download for the first time, and then when you modify java code to rebuild, both upload and download are fast (just upload and download a layer);
  3. In the development phase, the third method is the most convenient, but it is not appropriate if the kubernetes environment has multiple machines because the mirror has a local cache for the specified machine;

My kubernetes environment only has one computer, so I use method three with the following reference commands (it is recommended to install sshpass instead of entering the account password every time):

# Save the image as a tar file
sudo docker save bolingcavalry/probedemo:0.0.1 > probedemo.tar

# scp to kubernetes server
sshpass -p 888888 scp ./probedemo.tar root@192.168.50.135:/root/temp/202006/04/ 
  
# Execute ssh command remotely, load docker image
sshpass -p 888888 ssh root@192.168.50.135 "docker load < /root/temp/202006/04/probedemo.tar"

kubernetes deployment and service

  1. Create a name in kubernetes Probedemo.yamlThe file reads as follows, noting that the number of copies of the pod is 2, and also noting the parameter configurations for livenessProbe and readinessProbe:
apiVersion: v1
kind: Service
metadata:
  name: probedemo
spec:
  type: NodePort
  ports:
    - port: 8080
      nodePort: 30080
  selector:
    name: probedemo
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: probedemo
spec:
  replicas: 2
  template:
    metadata:
      labels:
        name: probedemo
    spec:
      containers:
        - name: probedemo
          image: bolingcavalry/probedemo:0.0.1
          tty: true
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
            initialDelaySeconds: 5
            failureThreshold: 10
            timeoutSeconds: 10
            periodSeconds: 5
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
            initialDelaySeconds: 5
            timeoutSeconds: 10
            periodSeconds: 5
          ports:
            - containerPort: 8080
          resources:
            requests:
              memory: "512Mi"
              cpu: "100m"
            limits:
              memory: "1Gi"
              cpu: "500m"
  1. Execute the command kubectl apply -f probedemo..yaml to create the deployment and service:

  1. The focus here is on the initial DelaySeconds and failureThreshold parameters of the liveness Probe. The initial DelaySeconds are equal to 5, which means that the pod is checked for survival probes after 5 seconds of creation. If the application does not complete start within 10 seconds and the survival probes do not return 200, they will be retried 10 times (failureThreshold equals 10). If the survival probes still fail to return 200 after 10 retries, the pod will be kubernetes kills and rebuilds. If it takes so long to start each time, the pod will be killed and rebuilt uninterruptedly.
  2. Execute the command kubectl apply -fProbedemo.yaml, create deployment and service, as shown in the following figure, the pod was created successfully in the tenth second, but it is not ready yet:

  1. Continue to view the status and after one minute the two pod s are finally ready:

  1. View the pod status with the kubectl describe command. Event notifications show that both the surviving and ready probes failed, but the status later becomes successful because of retries:

From coding to deployment, the probe technology for SpringBoot-2.3.0.RELEASE is verified.

Probe Technology for Verifying SpringBoot-2.3.0.RELEASE

  1. The purpose of the listening class AvailabilityListener is to listen for state changes, see the pod log, and see if the code for AvailabilityListener is valid. The following red box shows that AvailabilityListener was successfully callback during the application startup phase, printing the survive and ready state:

  1. The IP address of the machine on which kubernetes resides is 192.168.50.135, so the access address for the SpringBoot service is http://192.168.50.135:30080/xxx

  2. Access Address http://192.168.50.135:30080/actuator/health/liveness And the return code is shown in the red box below, so that the survival probe is turned on:

  1. The ready probe is also normal:

  1. Open both browsers and access: http://192.168.50.135:30080/hello , several times Ctrl+F5 brushes, as shown in the following figure, will quickly get different results, proving that the response comes from different Pod s:

  1. Visit: http://192.168.50.135:30080/statereader/get , you can get the status of survival and readiness, and you can see that StateReader's code is in effect, and you can get the status through the ApplicationAvailability interface:

  1. Modify the ready state to access: http://192.168.50.135:30080/statewriter/refuse , as shown in the red box below, you can see that the pod that received the request has had an exception in its ready state, proving that StateWritter.java After modifying the ready state in, kubernetes can be aware of this pod exception:

  1. Brushing the hello interface repeatedly with the browser returns only one Pod address, proving that only one Pod is responding to the request:

  1. Try to restore the service, note that the request is sent in the background of the server, and the IP address is to use the pod address that was just set to refuse:
curl http://10.233.90.195:8080/statewriter/accept
  1. As shown below, the state has been restored:

  1. Finally, try changing the survival state from CORRECT to BROKEN, browser access: http://192.168.50.135:30080/statewriter/broken
  2. As shown in the red box below, the number of restarts has changed to 1, indicating that the pod was killed once and is not currently ready due to the restart, proving that modifying the state of the survival probe in the SpringBoot triggered kubernetes to kill the pod:

  1. After waiting for the pod to restart and the ready probe to work properly, everything will be as it was before:

  1. Brush the browser strongly, as shown in the red box below, and both Pod s will respond properly:

Official Advice

  • So far, the "Master the Container Probe for SpringBoot-2.3" series has been completed. From theory to practice, we have learned the containerization technology that SpringBoot officially brought to us. Finally, we end with an official piece of advice that you will keep in mind:

  • My understanding of the above is: Be cautious when choosing services from external systems as probes (external systems may be databases or other web services). If there is a problem with external systems, it can cause kubernetes to kill pods (survival probe problem), or it can cause kubernetes to no longer schedule requests to pods (ready probe problem); (Thank you again for tolerating my EnglishHorizontal)

Welcome to my public number: programmer Xin Chen

https://github.com/zq2599/blog_demos

Tags: Java Kubernetes SpringBoot Docker

Posted on Wed, 10 Jun 2020 12:53:14 -0400 by ktsirig