JAVA application performance monitoring based on JDK command line tool monitoring

I. JVM parameter type

  1. JVM parameter type
  • Standard parameters
    -help
    -server -client
    -version -showversion
    -cp -classpath
    The standard parameters are basically unchanged and relatively stable in each version of the JVM.
  • X parameter
    Non standardized parameters may change in different versions of the JVM, but not much.
    -Xint: explain execution
    -Xcomp: compile to native code on first use
    -Xmixed: mixed mode, the JVM decides whether to compile into local code or not

  • XX parameter
    Non standardized parameters
    Relative instability
    Mainly used for JVM tuning and debugging
    Classification:

    Boolean type:
      Format: - XX: [+ -] < name > means to start or disable the name attribute
      For example: - XX:+UseConcMarkSweepGC
           -XX:+UseG1GC
     Non Boolean type:
      Format: - XX: < name > = < value > indicates that the value of the name attribute is value
      For example: - XX:MaxGCPauseMillis=500
           -XX:GCTimeRatio=19
      -Xmx -Xms: set JVM maximum and minimum memory
           -Xmx is equivalent to - XX:MaxHeapSize
           -Xms is equivalent to - XX:InitialHeapSize
           View through Jinfo - flag maxheapsize < PID >

    -xss is equivalent to - XX:ThreadStackSize thread stack size

    A thread is started by default, and its stack size is 1024kb

jstat viewing JVM statistics

options: -class, -compiler, -gc, -printcompilation 
  • Class loading

1000 10 means output 10 times every 1000ms
  • garbage collection

-gc, -gcutil, -gccasue, -gcnew, -gcold

Represents the usage of each partition of the current JVM memory, C represents the total capacity, and U represents the usage.
-gc output:
    S0C, S1C, S0U, S1U: total capacity and usage of S0 and S1
    EC, EU: total amount and usage of Eden District
    OC, OU: total amount and usage of Old area
    MC, MU: total amount and usage of Metaspace area
    CCSC, CCSU: total amount and usage of compressed space
    YGC, YGCT: number and times of YoungGC
    FGC, FGCT: number and time of FullGC
    GCT: total GC time
jstat -gc 23789 1000 10

JVM memory structure:

The non heap area is the local memory of the operating system, independent of the JVM heap area. JDK7 is called perm area, and JDK8 is called Metaspace.
CCS: when we enable short pointer, when we point to our own objects and class files, this CCS will exist. If we do not enable short pointer, this CCS will not exist.
CodeCache: the memory space of the native code generated by the JVM is called Code Cache; JIT compilation, JNI, etc. will compile the code to the native code, and the native code generated by JIT occupies most of the space of Code Cache.
Through jstat, you can view the relevant indicators of metaspace, which are M (Metaspace - Percent Used), CCS (Compressed Class Space - Percent Used), MC (Metaspace Capacity - Current), MU (Metaspae Used), CCSC (Compressed Class Space Capacity - Current), CCSU (Compressed Class Space Used), MCMN (Metaspace Capacity - Minimum), MCMX (metals Space capacity - maximum, CCSMN (Compressed Class Space Capacity - Minimum), CCSMX (Compressed Class Space Capacity - Maximum), the most important of which are the following four indicators (MC & MU & CCSC & CCSU):
  -MC indicates the total committed memory size of Klass Metaspace and NoKlass Metaspace. The unit is KB. Although we can see from the above definition that it is capacity, in essence, the calculation is not capacity, but committed, which should be noted.
 -MU this is not to blame, that is, the memory size used by Klass Metaspace and NoKlass Metaspace.
 -CCSC represents the commit ted memory size of Klass Metaspace, and the unit is KB
 -CCSU indicates the used memory size of Klass Metaspace
  • JIT compilation
    -compiler,-printcompilation

II. jmap+MAT actual combat memory overflow

Simulated memory overflow

Create the springboot project in spring.io and build it in Maven.
The project directory structure is:

The content of pom.xml is:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.1.7.RELEASE</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>zte.hdh</groupId>
    <artifactId>monitor_tuning</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>monitor_tuning</name>
    <description>Demo project for Spring Boot</description>

    <properties>
        <java.version>1.8</java.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>asm</groupId>
            <artifactId>asm</artifactId>
            <version>3.3.1</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>

Class User:

package zte.hdh.monitor_tuning.zdh.hdh.monitor_tuning.chapter2;

public class User {
    private int id;
    private String name;

    public User(){}

    public User(int id, String name) {
        this.id = id;
        this.name = name;
    }

    public int getId() {
        return id;
    }

    public void setId(int id) {
        this.id = id;
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }
}

Class Metaspace

package zte.hdh.monitor_tuning.zdh.hdh.monitor_tuning.chapter2;

import java.util.ArrayList;
import java.util.List;

import org.objectweb.asm.ClassWriter;
import org.objectweb.asm.MethodVisitor;
import org.objectweb.asm.Opcodes;

/**
 * https://blog.csdn.net/bolg_hero/article/details/78189621
 */
public class Metaspace extends ClassLoader {
    public static List<Class<?>> createClass() {
        // Class holding
        List<Class<?>> classes = new ArrayList<Class<?>>();
        // Cycle 1000w times to generate 1000w different classes.
        for (int i = 0; i < 10000000; ++i) {
            ClassWriter cw = new ClassWriter(0);
            // A class name is defined as Class{i}, its access domain is public, its parent class is java.lang.Object, and no interface is implemented
            cw.visit(Opcodes.V1_1, Opcodes.ACC_PUBLIC, "Class" + i, null,
                    "java/lang/Object", null);
            // Define constructor < init > method
            MethodVisitor mw = cw.visitMethod(Opcodes.ACC_PUBLIC, "<init>",
                    "()V", null, null);
            // The first instruction is to load this
            mw.visitVarInsn(Opcodes.ALOAD, 0);
            // The second instruction is to call the constructor of the parent class Object
            mw.visitMethodInsn(Opcodes.INVOKESPECIAL,"java/lang/Object",
                    "<init>", "()V");
            // The third instruction is return.
            mw.visitInsn(Opcodes.RETURN);
            mw.visitMaxs(1, 1);
            mw.visitEnd();

            Metaspace test = new Metaspace();
            byte[] code = cw.toByteArray();
            // Definition class
            Class<?> exampleClass = test.defineClass("Class" + i, code, 0, code.length);
            classes.add(exampleClass);
        }
        return classes;
    }
}

MemoryController class:

package zte.hdh.monitor_tuning.zdh.hdh.monitor_tuning.chapter2;

import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.ArrayList;
import java.util.List;
import java.util.UUID;

@RestController
public class MemoryController {

    private List<User> userList = new ArrayList<>();
    private List<Class<?>> classList = new ArrayList<>();

    /**
     * Heap memory overflow
     * -Xmx32M -Xms32M
     * @return
     */
    @GetMapping("/heap")
    public String heap(){
        int i = 0;
        while(true){
            userList.add(new User(i++, UUID.randomUUID().toString()));
        }
    }

    /**
     * Non heap memory overflow
     * -XX:MetaspaceSize=32M -XX:MaxMetaspaceSize=32M
     * @return
     */
    @GetMapping("/nonheap")
    public String noheap(){
        int i = 0;
        while(true){
            classList.addAll(Metaspace.createClass());
        }
    }
}

MonitorTuningApplication class:

package zte.hdh.monitor_tuning;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class MonitorTuningApplication {

    public static void main(String[] args) {
        SpringApplication.run(MonitorTuningApplication.class, args);
    }

}
Heap memory overflow, set the JVM parameter - Xmx32M -Xms32M, start the application, visit http://localhost:8080/heap, and the error is: Exception in thread "http-nio-8080-exec-1" java.lang.OutOfMemoryError: GC overhead limit exceeded.
Non heap memory overflow, set JVM parameter - XX:MetaspaceSize=32M -XX:MaxMetaspaceSize=32M, enable application, visit http://localhost:8080/nonheap, get error: Exception in thread "http-nio-8080-exec-1" java.lang.OutOfMemoryError: Metaspace.

How to export a memory image file

If it's a memory leak, we need to find out where the specific memory leak is, where it has not been released.
Memory leaks in Java and C + + are different. In C + +, memory leak means that after an object is new, the pointer of the object is lost, and this part of memory will never be released. In Java, after a new object is added, it occupies memory and is not released.

To export a memory image file:
  • Memory overflow auto export:

    -XX:+HeapDumpOnOutOfMemoryError
    -XX:HeapDumpPath=./
  • Use jmap command to export manually:

    jmap -dump.format=b,file=heap.hprof 16940
        

MAT analysis memory overflow

Memory Analyzer (MAT), Download: http://www.eclipse.org/mat/
There are two main statistics:

  • Number of objects

  • Object memory size

Regular expressions can be used to filter out the objects in our application for these two statistics, and the package name or object name can be used to filter them.
After selecting the object to be checked, right-click to select merge short path to GC roots - > exclude all phantom / week / soft etc. references to get the reference hierarchy of the object, as shown in the figure:

It's clear where the memory leak is.

3. jstack actual combat dead cycle and deadlock

jstack -options pid

JAVA thread status

Thread state

  • NEW: The thread has not yet started.
  • RUNNABLE: The thread is executing in the JVM.
  • BLOCKED: The thread is blocked waiting for a monitor lock.
  • WAITING: The thread is waiting indefinitely for another thread to perform a particular action.
  • TIMED_WAITING: The thread is waiting for another thread to perform an action for up to a specified waiting time.
  • TERMINATED: The thread has exited.

Our common BLOCKED status is to request locks and resources. For example, multithreading database operation, a more time-consuming operation, will cause other write operations to the database to be affected. For example, the operating system limits the number of file handles that can be opened. If the system has reached the threshold value, but has not closed properly, problems will occur.
Time  waiting is generally in the sleep method, wait method, join and other operations, waiting time.
RUNNABLE is our favorite thread state, which is working hard.

The thread states worthy of attention are:

  • Deadlock, deadlock (focus)
  • Running
  • Waiting on condition
  • Waiting on monitor entry
  • Suspended
  • Object.wait() or timed "waiting
  • Blocked (focus)
  • Stop, Parked

Jstack

What is jstck This is a tool that Oracle JDK contains by default to print the current thread stack information for executing Java processes.

jstack prints Java stack traces of Java threads for a given Java process or core file or a remote debug server. For each Java frame, the full class name, method name, 'bci' (byte code index) and line number, if available, are printed.

Several key points: the full class name and method name of each Java Frame. If you can get the line number, the line number will also be displayed. The information printed out by jstack is basically the same as the printStackTrace when an exception is encountered by a general application, except that it is only a thread call chain. Here, all threads in the application can be printed out by using jstack.
Usage:

Command name < optional parameter > + PID (process id)

Real combat dead cycle leads to high CPU

CpuController class:
`package zte.hdh.monitor_tuning.zdh.hdh.monitor_tuning.chapter2;

import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.ArrayList;
import java.util.List;

@RestController
public class CpuController {

@GetMapping("/loop")
public List<Long> loop(){
    String data = "{\"data\":[{\"partnerid\":]";
    return getPartneridsFromJson(data);
}

private static List<Long> getPartneridsFromJson(String data){
    //{\"data\":[{\"partnerid\":982,\"count\":\"10000\",\"cityid\":\"11\"},{\"partnerid\":983,\"count\":\"10000\",\"cityid\":\"11\"},{\"partnerid\":984,\"count\":\"10000\",\"cityid\":\"11\"}]}
    //It's normal data
    List<Long> list = new ArrayList<>(2);
    if(data == null || data.length() <= 0){
        return list;
    }
    int datapos = data.indexOf("data");
    if(datapos < 0){
        return list;
    }
    int leftBracket = data.indexOf("[",datapos);
    int rightBracket= data.indexOf("]",datapos);
    if(leftBracket < 0 || rightBracket < 0){
        return list;
    }
    String partners = data.substring(leftBracket+1,rightBracket);
    if(partners == null || partners.length() <= 0){
        return list;
    }
    while(partners!=null && partners.length() > 0){
        int idpos = partners.indexOf("partnerid");
        if(idpos < 0){
            break;
        }
        int colonpos = partners.indexOf(":",idpos);
        int commapos = partners.indexOf(",",idpos);
        if(colonpos < 0 || commapos < 0){
            //partners = partners.substring(idpos+"partnerid".length());//1
            continue;
        }
        String pid = partners.substring(colonpos+1,commapos);
        if(pid == null || pid.length() <= 0){
            //partners = partners.substring(idpos+"partnerid".length());//2
            continue;
        }
        try{
            list.add(Long.parseLong(pid));
        }catch(Exception e){
            //do nothing
        }
        partners = partners.substring(commapos);
    }
    return list;
}

}
`
Run jar, nohub Java - jar monitor - tuning-0.0.1-snapshot.jar
Browser access http://localhost:8080/loop
View CPU load, top
View jstack thread information, jstack 16108 > 16108.log
Monitor all threads of application, top-p 16108-h

printf "%x" 8247 => 2037
Query jstack log and find that multiple threads are calling CpuController.getPartneridsFromJson, and focus on troubleshooting this method.

Deadlock instance

CpuController class
`private Object lock1 = new Object();

private Object lock2 = new Object();
/**
 * deadlock
 * @return
 */
@RequestMapping("/deadlock")
public String deadlock(){
    new Thread(() -> {
        synchronized (lock1){
            try{
                Thread.sleep(1000);
            }catch (Exception e){
                e.printStackTrace();
            }
            synchronized (lock2){
                System.out.println("Thread1 over");
            }
        }
    }).start();
    new Thread(() -> {
        synchronized (lock2){
            try{
                Thread.sleep(1000);
            }catch (Exception e){
                e.printStackTrace();
            }
            synchronized (lock1){
                System.out.println("Thread2 over");
            }
        }
    }).start();
    return "deadLock";
}`

jstack print thread information:

Tags: Java jvm Spring Maven

Posted on Sat, 09 Nov 2019 11:05:02 -0500 by lewel