Principle Analysis of Ali Monitoring and Diagnostic Tool Arthas Source Code

Previously, Ali opened its source monitoring and diagnostics tool, Arthas, which is a useful tool for online problem analysis and has received a lot of attention in the short term. It's great that even Java's official Twitter has been forwarded on Twitter.

This is what GitHub says:

Arthas is an on-line monitoring diagnostic product that can view the status information of application loads, memory, gc, threads in real time from a global perspective, and diagnose business problems without modifying the application code, including viewing the incoming and outgoing parameters of method calls, exceptions, time-consuming monitoring method execution, class loading information, etc. which greatly improves the efficiency of on-line problem detection.

I often see open source tools of interest and look for a few of the most interesting features to jump in and understand the design and implementation principles from the source code.For some of the implementation ideas that you know, and then verify from the source whether the same implementation ideas are used.If you achieve what you think, you might think, oh, think together.If there's another implementation in the source code, you'll think about Cool and you can play like that.It's like talking to the author of the source code.

This time I took advantage of the National Day holidays to see some "Arthas" source code, to summarize roughly.

From the package structure of the source code, you can see that there are several large modules:

  • Agent - Custom Agent loaded by VM
  • Client-Telnet Client Implementation
  • Core-Arthas core implementation, including connecting VM s, parsing commands, etc.
  • Site - Arthas Help Manual Site Content

I've looked at the following features:

  • Connection process
  • Decompile class, get source code
  • Query specified loaded class

Connection process

Connecting to a specified process is the basis for subsequent monitoring and diagnosis.Only attach to the process first can you get information about the VM, query the classes loaded by ClassLoader, and so on.

How do I connect to a process?

Readers of diagnostic tools like JProfile and VisualVM will probably be impressed with the fact that tools like JProfile and VisualVM let you choose a process to connect to.It then operates on the specified VM.Examples include viewing the corresponding memory partition information, memory garbage collection information, executing BTrace scripts, and so on.

Let's first think about how these lists of processes can be connected.

Typically, this might be something like ps aux | grep java, or using the Java-provided tool jps-lv, you can list the content that contains the process id.I wrote a little bit about JPS in an earlier article. Several java gadgets that you may not know ), which implements that all locally started Java processes are stored in a temporary Java directory with pid as the file name.This list can be obtained by traversing these files.

What does Arthas do?

In Startup Script as.sh In, the code for the process list is as follows, which is also implemented by JPS and then excludes the Jps themselves:

# check pid
    if [ -z ${TARGET_PID} ] && [ ${BATCH_MODE} = false ]; then
        local IFS_backup=$IFS
        IFS=$'\n'
        CANDIDATES=($(${JAVA_HOME}/bin/jps -l | grep -v sun.tools.jps.Jps | awk '{print $0}'))

        if [ ${#CANDIDATES[@]} -eq 0 ]; then
            echo "Error: no available java process to attach."
            # recover IFS
            IFS=$IFS_backup
            return 1
        fi

        echo "Found existing java process, please choose one and hit RETURN."

        index=0
        suggest=1
        # auto select tomcat/pandora-boot process
        for process in "${CANDIDATES[@]}"; do
            index=$(($index+1))
            if [ $(echo ${process} | grep -c org.apache.catalina.startup.Bootstrap) -eq 1 ] \
                || [ $(echo ${process} | grep -c com.taobao.pandora.boot.loader.SarLauncher) -eq 1 ]
            then
               suggest=${index}
               break
            fi
        done
  •  

Once the process is selected, you are connected to the specified process.Connection section is attach ed here

# attach arthas to target jvm
# $1 : arthas_local_version
attach_jvm()
{
    local arthas_version=$1
    local arthas_lib_dir=${ARTHAS_LIB_DIR}/${arthas_version}/arthas

    echo "Attaching to ${TARGET_PID} using version ${1}..."

    if [ ${TARGET_IP} = ${DEFAULT_TARGET_IP} ]; then
        ${JAVA_HOME}/bin/java \
            ${ARTHAS_OPTS} ${BOOT_CLASSPATH} ${JVM_OPTS} \
            -jar ${arthas_lib_dir}/arthas-core.jar \
                -pid ${TARGET_PID} \
                -target-ip ${TARGET_IP} \
                -telnet-port ${TELNET_PORT} \
                -http-port ${HTTP_PORT} \
                -core "${arthas_lib_dir}/arthas-core.jar" \
                -agent "${arthas_lib_dir}/arthas-agent.jar"
    fi
}
  •  

For attach implementations inside the JVM, they are implemented through com.sun.tools.attach.VirtualMachine and VirtualMachine.attach(pid) in tools.jar.

The bottom layer is via JVMTI.Previous articles briefly analyzed JVMTI as a technology ( When we talk about Debug, what are we talking about (Debug implementation principles) ) Load the custom Agent and communicate with the VM before or at run time.

The specific implementation above is in the main class of arthas-core.jar, so let's look at it:

private void attachAgent(Configure configure) throws Exception {
        VirtualMachineDescriptor virtualMachineDescriptor = null;
        for (VirtualMachineDescriptor descriptor : VirtualMachine.list()) {
            String pid = descriptor.id();
            if (pid.equals(Integer.toString(configure.getJavaPid()))) {
                virtualMachineDescriptor = descriptor;
            }
        }
        VirtualMachine virtualMachine = null;
        try {
            if (null == virtualMachineDescriptor) { // Use attach(String pid) this way
                virtualMachine = VirtualMachine.attach("" + configure.getJavaPid());
            } else {
                virtualMachine = VirtualMachine.attach(virtualMachineDescriptor);
            }

            Properties targetSystemProperties = virtualMachine.getSystemProperties();
            String targetJavaVersion = targetSystemProperties.getProperty("java.specification.version");
            String currentJavaVersion = System.getProperty("java.specification.version");
            if (targetJavaVersion != null && currentJavaVersion != null) {
                if (!targetJavaVersion.equals(currentJavaVersion)) {
                    AnsiLog.warn("Current VM java version: {} do not match target VM java version: {}, attach may fail.",
                                    currentJavaVersion, targetJavaVersion);
                    AnsiLog.warn("Target VM JAVA_HOME is {}, try to set the same JAVA_HOME.",
                                    targetSystemProperties.getProperty("java.home"));
                }
            }

            virtualMachine.loadAgent(configure.getArthasAgent(),
                            configure.getArthasCore() + ";" + configure.toString());
        } finally {
            if (null != virtualMachine) {
                virtualMachine.detach();
            }
        }
    }
  •  

With VirtualMachine, you can attach to the currently specified pid, or attach to a specified process through VirtualMachine Descriptor. The core of this sentence is:

virtualMachine.loadAgent(configure.getArthasAgent(),configure.getArthasCore() + ";" + configure.toString());
  • 1

In this way, a connection is established with the VM of the specified process, and communication is now possible.

Decompilation implementation of classes

In problem diagnosis, sometimes we need to know what the currently loaded classes correspond to, to confirm whether the loaded classes are correct, etc. Generally, javap can only display summary-like content, which is not intuitive.On the desktop, we can use tools like jd-gui, but there are generally not many options on the command line.Arthas integrates this functionality.

The general steps are as follows:

  • Find the class first by specifying the contents of the class name
  • Depending on the option, determine whether a lookup such as Inner Class will occur
  • Decompile

Let's look at the implementation of Arthas.
For a class lookup with a specified name in the VM, let's look at the following lines of code:

    public void process(CommandProcess process) {
        RowAffect affect = new RowAffect();
        Instrumentation inst = process.session().getInstrumentation();
        Set<Class> matchedClasses = SearchUtils.searchClassOnly(inst, classPattern, isRegEx, code);

        try {
            if (matchedClasses == null || matchedClasses.isEmpty()) {
                processNoMatch(process);
            } else if (matchedClasses.size() > 1) {
                processMatches(process, matchedClasses);
            } else {
                Set<Class> withInnerClasses = SearchUtils.searchClassOnly(inst,  classPattern + "(?!.*\\$\\$Lambda\\$).*", true, code);
                processExactMatch(process, affect, inst, matchedClasses, withInnerClasses);
    }
  •  

The key lookup content, encapsulated, has a core parameter in SearchUtils: Instrumentation, which is implemented by this buddy.

    /**
     * Search for classes that have been loaded by the JVM based on class name matching
     *
     * @param inst             inst
     * @param classNameMatcher Class name matching
     * @return Matched Class Set
     */
    public static Set> searchClass(Instrumentation inst, Matcher classNameMatcher, int limit) {
        for (Class clazz : inst.getAllLoadedClasses()) {
            if (classNameMatcher.matching(clazz.getName())) {
                matches.add(clazz);
            }
        }
        return matches;
    }
  •  

inst.getAllLoadedClasses(), which is the big player behind it.

How do I decompile a Class once it's found?

 private String decompileWithCFR(String classPath, Class clazz, String methodName) {
        List<String> options = new ArrayList<String>();
        options.add(classPath);
//        options.add(clazz.getName());
        if (methodName != null) {
            options.add(methodName);
        }
        options.add(OUTPUTOPTION);
        options.add(DecompilePath);
        options.add(COMMENTS);
        options.add("false");
        String args[] = new String[options.size()];
        options.toArray(args);
        Main.main(args);
        String outputFilePath = DecompilePath + File.separator + Type.getInternalName(clazz) + ".java";
        File outputFile = new File(outputFilePath);
        if (outputFile.exists()) {
            try {
                return FileUtils.readFileToString(outputFile, Charset.defaultCharset());
            } catch (IOException e) {
                logger.error(null, "error read decompile result in: " + outputFilePath, e);
            }
        }

        return null;
    }
  • By following this approach: decompileWithCFR, we probably understand that decompiling is achieved through a third-party tool, CFR.The above code is also spelled Option and passed to the CFR's Main method implementation, which is then saved.Interested friends can consult benf cfr for specific usage.

Implementation of Query Loading Class

After looking at the contents of the decompiled class above, we know that it encapsulates a SearchUtil class, which is used in many places later, and the decompilation above is done after the class is queried.The process of query is also based on Instrument, plus various matching rule filtering, so more specific content will not be described.

We found two key things in the implementation of the above functions:

  • VirtualMachine
  • Instrumentation

The overall logic of Arthas is also based on Java Instrumentation. All the classes loaded are loaded by Agent, enhanced by addTransformer, and then weaved into the corresponding Advice. SearchUtil is used to find the classes and methods, and the loadAllClass method of Instrumentation is used to load the CL of all JVM s.Ass matches by name, and consistent returns.

Instrumentation is a good comrade!What?

Tags: Programming Java jvm github Tomcat

Posted on Sat, 09 Nov 2019 01:55:38 -0500 by gijs