JVM - 15. Locate out-of-heap memory OOM

2.1 Out-of-heap memory request

ByteBuffer byteBuffer = ByteBuffer.allocateDirect(1 * 1024 * 1024);

public static ByteBuffer allocateDirect(int capacity) { return new DirectByteBuffer(capacity); }

When memory is requested using ByteBuffer's static method allocateDirect(), a DirectByteBuffer object is created using the construction method of the DirectByteBuffer class:

DirectByteBuffer(int cap) { super(-1, 0, cap, cap); boolean pa = VM.isDirectMemoryPageAligned(); int ps = Bits.pageSize(); long size = Math.max(1L, (long)cap + (pa ? ps : 0)); // Determine if there is enough room for an application // size: The amount of memory that you get to actually request, depending on whether or not you align the pages // cap: User specifies required memory size (<=size) Bits.reserveMemory(size, cap); long base = 0; try { // Call UNsafe method to request memory base = unsafe.allocateMemory(size); } catch (OutOfMemoryError x) { // Failed request, free memory Bits.unreserveMemory(size, cap); throw x; } // Initial memory space is 0 unsafe.setMemory(base, size, (byte) 0); if (pa && (base % ps != 0)) { // Round up to page boundary address = base + ps - (base & (ps - 1)); } else { address = base; } // Registering memory recycling handlers using the Cleaner mechanism cleaner = Cleaner.create(this, new Deallocator(base, size, cap)); att = null; }

When using a construction method to request memory, first determine if there is enough memory space to apply for, and if there is space to apply, update the corresponding variables; If not enough to call GC first, throw OOM if not enough;

static void reserveMemory(long size, int cap) { if (!memoryLimitSet && VM.isBooted()) { // Gets the maximum amount of external memory that can be requested, default is 64MB // This size can be set by the jvm parameter -XX:MaxDirectMemorySize=<size> maxMemory = VM.maxDirectMemory(); memoryLimitSet = true; } // optimist! // Try to reserve space and return to true if there is enough space if (tryReserveMemory(size, cap)) { return; } final JavaLangRefAccess jlra = SharedSecrets.getJavaLangRefAccess(); // retry while helping enqueue pending Reference objects // which includes executing pending Cleaner(s) which includes // Cleaner(s) that free direct buffer memory // Maybe Cleaner will free up space, try again, and continue trying to reserve space while (jlra.tryHandlePendingReference()) { if (tryReserveMemory(size, cap)) { return; } } // trigger VM's Reference processing // Not enough, call the GC here to do a space reclaim, freeing up references to out-of-heap memory System.gc(); // a retry loop with exponential back-off delays // (this gives VM some time to do it's job) // Continue trying to reserve space through a delayed loop with sleep and return if the reservation is successful; // If MAX_is exceeded The number of SLEEPS was not successful, throwing a Direct buffer memory exception directly boolean interrupted = false; try { long sleepTime = 1; int sleeps = 0; while (true) { if (tryReserveMemory(size, cap)) { return; } if (sleeps >= MAX_SLEEPS) { break; } if (!jlra.tryHandlePendingReference()) { try { Thread.sleep(sleepTime); sleepTime <<= 1; sleeps++; } catch (InterruptedException e) { interrupted = true; } } } // no luck // Throw OOM if several retries are not enough throw new OutOfMemoryError("Direct buffer memory"); } finally { if (interrupted) { // don't swallow interrupts Thread.currentThread().interrupt(); } } }

After the implementation of this method, that is, when there is room to apply, the application space starts:

try { base = unsafe.allocateMemory(size); } catch (OutOfMemoryError x) { Bits.unreserveMemory(size, cap); throw x; }

allocateMemory() This method requests memory by unsafe through a JNI call to C's malloc;

2.2 Out-of-heap memory release

After the above request for out-of-heap memory, a Cleaner memory recycling object was registered:

cleaner = Cleaner.create(this, new Deallocator(base, size, cap));

This is because the DirectByteBuffer object itself is in the heap and is normally recycled by the GC of the jvm; However, the out-of-heap memory region associated with DirectByteBuffer will not be reclaimed by GC, so a mechanism is needed to reclaim the memory it requested outside the heap when DirectByteBuffer is reclaimed.

When an object is recycled, java provides two features to do additional work:

• Implement some custom actions in finalize() before objects are recycled (Java official does not recommend);
• Handles custom actions after objects are recycled through virtual references (objects with virtual references receive a system notification when they are recycled by the GC);

In the release of out-of-heap memory here, java uses a virtual reference method; A Cleaner class is provided to simplify these operations. Cleaner is a subclass of PhantomReference that triggers the corresponding Runnable callback method when PhantomReference is joined to the Reference Queue queue.

new Deallocator(base, size, cap);

Here Deallocator is a class that implements the Runnable interface:

private static class Deallocator implements Runnable { private static Unsafe unsafe = Unsafe.getUnsafe(); private long address; private long size; private int capacity; private Deallocator(long address, long size, int capacity) { assert (address != 0); this.address = address; this.size = size; this.capacity = capacity; } public void run() { if (address == 0) { // Paranoia return; } unsafe.freeMemory(address); address = 0; Bits.unreserveMemory(size, capacity); } }

You can see that it overrides the run() method to call the unsafe method to free up memory and reset the statistical variables;

So DirectByteBuffer uses the Cleaner mechanism to create a PhantomReference by treating itself as a dummy reference object. So when it is GC itself, it is added to the reference queue;

This reference queue is then monitored to call the run() method of the Deallocator class that implements the Runnable interface on the objects inside it to free up the out-of-heap memory.

From the above description, we know the specific implementation of allocation and recycling of out-of-heap memory, so what happens when out-of-heap memory is not enough, even OOM?

1. If the system instantaneously accepts a large number of concurrent requests, creates a large number of DirectByteBuffer s, and is slow to process, and cannot be removed by GC in time, the natural out-of-heap memory will not be released, and then the out-of-heap memory will overflow;
   
2. Because the jvm memory area of the system is not reasonably divided, after the YGC of Eden area occurs, there will be no survivors in Survivor area and they will enter the old age.
   • Normally, however, FGC occurs very often in older generations, that is, DirectByteBuffer objects entering older generations are not recycled for a long time, natural out-of-heap memory is not released, and then out-of-heap memory overflows.
3. For the second case, the jvm actually takes into account; So it will call System.gc() once to trigger the GC manually when it requests insufficient memory.
   • However, if you use the -XX:+DisableExplicitGC parameter to turn off the display GC;
   • Therefore, if the display GC is turned off using the -XX:+DisableExplicitGC parameter, the second case will result in an out-of-heap memory overflow;

4.1 Simulation 1

/** * * Direct memory overflow * * jvm options: * -XX:MaxDirectMemorySize=100m -verbose:gc -XX:+PrintGCDetails */ public class DirectByteBufferOomDemo { public static void main(String[] args) { int count = 0; List<ByteBuffer> list = new ArrayList<>(); while (true) { ByteBuffer byteBuffer = ByteBuffer.allocateDirect(1 * 1024 * 1024); list.add(byteBuffer); System.out.println("Currently created " + (++count) + "M Object"); } } }

The code is simple, just in a dead loop, you always apply for a DirectctByteBuffer object of size 1M, and then you join a list, that is, a list that always holds its references and will not be dropped by the GC;

Then use -XX:MaxDirectMemorySize=100m to specify the maximum out-of-heap memory value of 100M, so the out-of-heap memory OOM should occur when the request is almost 100M;

Look at the implementation:

You can see that 100M objects are generated here, and when the generation continues, the out-of-heap memory area is not enough, then System.gc() is executed first;
However, since object reference holdings cannot be removed by GC, a java.lang.OutOfMemoryError: Direct buffer memory occurs.

4.2 Simulation 2

/** * * Direct memory overflow * * jvm options: * -XX:MaxDirectMemorySize=100m -verbose:gc -XX:+PrintGCDetails */ */ public class DirectByteBufferOomDemo { public static void main(String[] args) { int count = 0; // List<ByteBuffer> list = new ArrayList<>(); while (true) { ByteBuffer byteBuffer = ByteBuffer.allocateDirect(1 * 1024 * 1024); // list.add(byteBuffer); System.out.println("Currently created " + (++count) + "M Object"); } } }

Unlike simulation 1, comment out the list's code so that the requested DirectctByteBuffer object does not have a reference point, that is, it becomes a garbage object;

Out-of-heap memory OOM s should not occur here because they are recycled when System.gc() is called manually in DirectByteBuffer when out-of-heap memory is requested;

Look at the implementation:

You can see that System.gc() is executed every time a 100M object is generated;
Since objects are garbage objects that can be recycled and continue to be generated after GC, there is no java.lang.OutOfMemoryError: Direct buffer memory;

4.3 Simulation 3

/** * * Direct memory overflow * * jvm options: * -XX:MaxDirectMemorySize=100m -verbose:gc -XX:+PrintGCDetails -XX:+DisableExplicitGC */ public class DirectByteBufferOomDemo { public static void main(String[] args) { int count = 0; // List<ByteBuffer> list = new ArrayList<>(); while (true) { ByteBuffer byteBuffer = ByteBuffer.allocateDirect(1 * 1024 * 1024); // list.add(byteBuffer); System.out.println("Currently created " + (++count) + "M Object"); } } }

Unlike simulation 2, the jvm parameter -XX:+DisableExplicitGC was added, which prevents the System.gc() operation from being displayed in the code.
Therefore, when the out-of-heap memory is full, the execution of System.gc() is invalid, the DirectByteBuffer object that is already a garbage object cannot be recycled, and the out-of-heap memory cannot be freed, and an out-of-heap memory OOM should occur.

Look at the implementation:

You can see that when a 100M object is generated, java.lang.OutOfMemoryError: Direct buffer memory occurs;

Similarly, when you encounter a system crash, you should first log on to the server to check the system log. You will see an error like java.lang.OutOfMemoryError: Direct buffer memory, which is OOM of out-of-heap memory.

When an out-of-heap memory OOM occurs, even if you configure the dump switch, it will not generate a dump log when the system crashes (obviously, there is no out-of-heap memory in the heap, and it certainly will not generate a dump log for you);

But underneath this error, there will be specific classes and methods that actually trigger out-of-heap memory OOM and the number of lines of code inside.
This way you know if you are manually generating out-of-heap memory in your code or if you are generating out-of-heap memory in some frameworks, such as jetty, netty, and so on.

• If this is the result of manually generating out-of-heap memory in your code, then you need to carefully check if your code has any problems, if too many DirectByteBuffer s have been generated and cannot be recycled.
• If your code doesn't have this problem, or if it's a problem raised within the framework, you have to think about it from the GC:
  • First check to see if -XX:+DisableExplicitGC is configured;
    • If configured, you must release this restriction for System.gc() in DirectByteBuffer to take effect;
    • However, at this point, you should be aware of your own code and try not to have System.gc() executed, otherwise it may trigger frequent FGC;
    • If this parameter is configured, then removing this configuration will basically solve the problem.
  • If not, it's a bit complicated; You need to use the jstat tool we described earlier to determine if there is a problem with jvm memory area size allocation.
    • For example, Above In the case mentioned, the DirectByteBuffer object has entered an older age, delaying its recycling and failing to free up out-of-heap memory space, resulting in out-of-heap memory OOM;
    • When such problems are detected, the jvm parameters need to be adjusted to appropriately allocate the memory size of each region. Make it possible for DirectByteBuffer objects to be recycled in time to free up memory outside the heap;
    • The specific way to see if the jvm memory partitioning is reasonable is not described here again, you can refer to the previous articles.

Add new comment

0 comments