Source file PF4J cannot be deleted after class uninstallation

background

We have a Plugin management system that enables hot loading of Jar packages based on a Plugin management class library PF4J , similar to OSGI, is now a thousand Star project on GitHub. The following is an official description of the library > A plugin is a way for a third party to extend the functionality of an application. A plugin implements extension points declared by application or other plugins. Also a plugin can define extension points. With PF4J you can easily transform a monolithic java application in a modular application.

In general, PF4J can dynamically load Class files.It also enables dynamic uninstallation of Class files.

Problem Description

There is a new requirement to hot update the version of Plugin.That is, ubload the old Plugin version that has already been loaded into the JVM, then load the new version of Plugin.PF4J works well.To prevent too many expired Plugins, each update deletes the old version.However, something strange happened: > - Calls the File.delete() method to return true, but the old file is still there > - Delete files manually and report process usage errors > - When the JVM exits at the end of the program, the file is gone

Here is a simple test code based on PF4j version 3.0.1:

public static void main(String[] args) throws InterruptedException {
    // create the plugin manager
    PluginManager pluginManager = new DefaultPluginManager();
    // start and load all plugins of application
    Path path = Paths.get("test.jar");
    pluginManager.loadPlugin(path);
    pluginManager.startPlugins();

    // do something with the plugin

    // stop and unload all plugins
    pluginManager.stopPlugins();
    pluginManager.unloadPlugin("test-plugin-id");
    try {
        // There are no errors here
        Files.delete(path);
    } catch (IOException e) {
        e.printStackTrace();
    }

    // The file persists until it is automatically deleted after the 5s-minute program exits
    Thread.sleep(5000);
}

I went to google for a circle and got nothing, but in Issues of PF4J project, someone reported the same Bug But it's not going to be Close anymore.

Problem Location

It seems I can only solve it by myself. From the code above, you can see that the Plugin management of PF4J is operated through the PluginManager class.This class defines a series of operations: getPlugin(), loadPlugin(), stopPlugin(), unloadPlugin()...

unloadPlugin

The core code is as follows:

private boolean unloadPlugin(String pluginId) {
    try {
        // Set Plugin to Stop state
        PluginState pluginState = this.stopPlugin(pluginId, false);
        if (PluginState.STARTED == pluginState) {
            return false;
        } else {
            // Get the wrapper class (proxy class) of Plugin and think of it as the Plugin class
            PluginWrapper pluginWrapper = this.getPlugin(pluginId);
            // Delete various references to this Plugin in PluginManager for GC convenience
            this.plugins.remove(pluginId);
            this.getResolvedPlugins().remove(pluginWrapper);
            // Event triggering unload
            this.firePluginStateEvent(new PluginStateEvent(this, pluginWrapper, pluginState));
            // A consistent style of hot deployment where a Jar has a ClassLoader:Map whose Key is PluginId and Value is the corresponding ClassLoader
            // ClassLoader is custom, called PluginClassLoader
            Map<string, classloader> pluginClassLoaders = this.getPluginClassLoaders();
            if (pluginClassLoaders.containsKey(pluginId)) {
                // Delete the reference to ClassLoader for GC convenience
                ClassLoader classLoader = (ClassLoader)pluginClassLoaders.remove(pluginId);
                if (classLoader instanceof Closeable) {
                    try {
                        // Give ClassLoader away to close, freeing up all resources
                        ((Closeable)classLoader).close();
                    } catch (IOException var8) {
                        throw new PluginRuntimeException(var8, "Cannot close classloader", new Object[0]);
                    }
                }
            }

            return true;
        }
    } catch (IllegalArgumentException var9) {
        return false;
    }
}

public class PluginClassLoader extends URLClassLoader {
}

The code logic is straightforward and is the standard process of unloading a Class: leaving the reference to Plugin empty and then dropping the corresponding ClassLoader close to free up resources.It is important to note here that this ClassLoader is a subclass of URLClassLoader, which implements the Closeable interface and frees up resources for any confusion. This Article. Class unload section, I don't see any problems for the moment.

loadPlugin

The loading of Plugin is a bit more complex, with the core logic as follows

protected PluginWrapper loadPluginFromPath(Path pluginPath) {
    // Get PluginDescriptorFinder to find PluginDescriptor
    // There are two types of Finder s, one through Manifest and one through the properties file
    // It is conceivable that there will be IO reads here
    PluginDescriptorFinder pluginDescriptorFinder = getPluginDescriptorFinder();
    // Find PluginDescriptor through PluginDescriptorFinder
    // PluginDescriptor records a series of information such as Plugin Id, Plugin name, PluginClass, etc.
    // This is simply loading information about plugins configured in Java Manifest or in the plugin.properties file
    PluginDescriptor pluginDescriptor = pluginDescriptorFinder.find(pluginPath);

    pluginId = pluginDescriptor.getPluginId();
    String pluginClassName = pluginDescriptor.getPluginClass();

    // Load Plugin
    ClassLoader pluginClassLoader = getPluginLoader().loadPlugin(pluginPath, pluginDescriptor);
    // Create a wrapper class (proxy) for Plugin that contains all the information about Plugin
    PluginWrapper pluginWrapper = new PluginWrapper(this, pluginDescriptor, pluginPath, pluginClassLoader);
    // Set up a Plugin creation factory, and subsequent instances of Plugin are created through factory mode
    pluginWrapper.setPluginFactory(getPluginFactory());

    // Some validation
    ......

    // Cache the loaded Plugin
    // The unloadPlugin operation described above may correspond to the
    plugins.put(pluginId, pluginWrapper);
    getUnresolvedPlugins().add(pluginWrapper);
    getPluginClassLoaders().put(pluginId, pluginClassLoader);

    return pluginWrapper;
}

There are four more important classes > 1. PluginDescriptor: The class used to describe Plugin.A PF4J's Plugin must identify Plugin information in Jar's Manifest(pom's "manifestEntries" or "MANIFEST.MF" file), such as entry Class, PluginId, Plugin Version, and so on. > 2. PluginDescriptorFinder: The tool class used to find PluginDescriptor has two implementations by default: Manifest PluginDescriptorFinder and Proerties PluginDescriptorFinder, which, as the name implies, corresponds to two ways of finding Plugin information. > 3. A wrapper class for PluginWrapper:Plugin that holds references to Plugin instances and provides access to corresponding information such as PluginDescriptor (ClassLoader). > 4. PluginClassLoader: A custom class loader that inherits from URLClassLoader and overrides the **loadClass()** method to achieve the target Plugin load.

Looking back at the problem at the beginning, deletion of files is usually caused by other process usage, and the file stream is not deleted in time after it is opened.But we checked that all the file stream operations that occurred during the above process had Close.It seems like we're in a deadlock.

MAT

Change your mind, since you can't delete files, you'll see what's inside the JVM. Run the test code, then look up the Java process ID (in this case, 11210) with the command jps, and dump out the alive object in the JVM to a file tmp.bin with the following command: > jmap -dump:live,format=b,file=tmp.bin 11210

Next, open the dump file in the memory analysis tool MAT, and the result is as follows:

A class, com.sun.nio.zipfs.ZipFileSystem, was found to account for the majority (68.8%) of all references held by sun.nio.fs.WindowsFileSystemProvider.Based on this clue, we go into the code to see where there are api calls to FileSystem and, of course, we found a behind-the-scenes black hand in PropertiesPluginDescriptorFinder (just keep the core code):

/**
 * Find a plugin descriptor in a properties file (in plugin repository).
 */
public class PropertiesPluginDescriptorFinder implements PluginDescriptorFinder {
    // Call this method to find plugin.properties and load Plugin-related information
    public PluginDescriptor find(Path pluginPath) {
        // Focus on the getPropertiesPath method
        Path propertiesPath = getPropertiesPath(pluginPath, propertiesFileName);

        // Read properties file contents
        ......

        return createPluginDescriptor(properties);
    }
	
    protected Properties readProperties(Path pluginPath) {
        Path propertiesPath;
        try {
            // The file eventually gets the Path variable from the tool class FileUtils
            propertiesPath = FileUtils.getPath(pluginPath, propertiesFileName);
        } catch (IOException e) {
            throw new PluginRuntimeException(e);
        }
        
        // Load properties file
        ......
        return properties;
    }
}

public class FileUtils {
    public static Path getPath(Path path, String first, String... more) throws IOException {
        URI uri = path.toUri();
        // Initialization of other variables, skip
		......
		
        // FileSystem appears as the culprit when loading Path through FileSystem!!!
        // After getting FileSystem here, there are no closed resources!!!
        // Hide too deep
        return getFileSystem(uri).getPath(first, more);
    }
	
    // This method returns a FileSystem instance, note that method signatures have IO operations
    private static FileSystem getFileSystem(URI uri) throws IOException {
        try {
            return FileSystems.getFileSystem(uri);
        } catch (FileSystemNotFoundException e) {
            // If uri does not exist, return an empty FileSystem bound to this uri
            return FileSystems.newFileSystem(uri, Collections.<string, string>emptyMap());
        }
    }
}

At last, it corresponded to the analysis result of MAT.The original PropertiesPluginDescriptorFinder loaded the Plugin description through FileSystem, but after loading, the FileSystem.close() method was not called to release the resource.DefaultPluginManager used in our project contains two DescriptorFinders by default:

    protected PluginDescriptorFinder createPluginDescriptorFinder() {
        // DefaultPluginManager's PluginDescriptorFinder is a List
        // Using combination mode, PluginDescriptor is loaded in the order in which it was added
        return new CompoundPluginDescriptorFinder()
            // Add PropertiesPluginDescriptorFinder to List
            .add(new PropertiesPluginDescriptorFinder())
            // Add ManifestPluginDescriptorFinder to List
            .add(new ManifestPluginDescriptorFinder());
    }

Ultimately, the ManifestPluginDescriptorFinder is actually used, but the code first loads it once with PropertiesPluginDescriptorFinder (holding a reference to the file regardless of whether the load is successful or not), finds it cannot be loaded, and then uses ManifestPluginDescriptorFinder.This explains that when the JVM exits, the file is automatically deleted because the resource is forcibly released.

Problem solving

Write your own class that inherits PropertiesPluginDescriptorFinder, override the readProperties() method, call your own MyFileUtil.getPath() method, and after using FileSystem.getPath, remove the FileSystem close with the following core code:

public class FileUtils {
    public static Path getPath(Path path, String first, String... more) throws IOException {
        URI uri = path.toUri();
        ......
        // When finished, call FileSystem.close()
        try (FileSystem fs = getFileSystem(uri)) {
            return fs.getPath(first, more);
        }
    }
    
    private static FileSystem getFileSystem(URI uri) throws IOException {
        try {
            return FileSystems.getFileSystem(uri);
        } catch (FileSystemNotFoundException e) {
            return FileSystems.newFileSystem(uri, Collections.<string, string>emptyMap());
        }
    }
}

Follow-up

A bug that's been hidden so far... It's not a big problem, but it's been a real problem for us for some time, and it's true that some of our colleagues have had similar problems.Publishing PR to PF4J to solve this stubborn problem is also a small contribution to the open source community in order to prevent subsequent classmates from encountering similar situations.

summary

Files cannot be deleted, 95% of the time because the resource is not released cleanly. PF4J can load Plugin's description information in two ways, one from the configuration file plugin.progerties and the other from Manifest.The default behavior is to load through plugin.progerties first, and Manifest if it cannot. The method loaded through plugin.progerties is internally implemented through nio's FileSystem.However, after loading through FileSystem and until Plugin unload, no **FileSystem.close()** method is called to release resources, resulting in a bug that the file cannot be deleted.

FileSystem was created by FileSystemProvider, which has different implementations under different systems.As the implementation under Windows is as follows:

FileSystemProvider is cached after it is created as a static member variable of the tool class FIleSystems, so FileSystemProvider will not be GC.Whenever the FileSystemProvider creates a FileSystem, it caches the FileSystem in one of its Map s, so normally FileSystem will not be GC, just like the analysis done by MAT above.FileSystem's close() method, one of which is to release references, allows the class to be reclaimed from memory, the resource to be released, and the file to be deleted normally after closing

public class ZipFileSystem extends FileSystem {
    // FileSystem's own provider
    private final ZipFileSystemProvider provider;
    public void close() throws IOException {
        ......
        // Remove your own reference from the provider
        this.provider.removeFileSystem(this.zfpath, this);
        ......
    }
}

public class ZipFileSystemProvider extends FileSystemProvider {
    // This Map saves all FileSystem s created by this Provider
    private final Map<path, zipfilesystem> filesystems = new HashMap();

    void removeFileSystem(Path zfpath, ZipFileSystem zfs) throws IOException {
        // Really delete references
        synchronized(this.filesystems) {
            zfpath = zfpath.toRealPath();
            if (this.filesystems.get(zfpath) == zfs) {
                this.filesystems.remove(zfpath);
            }

        }
    }
}
~~~</path,></string,></string,></string,>

Tags: Programming jvm Java github hot update

Posted on Wed, 11 Dec 2019 21:51:30 -0500 by gateway69