First investigation of OOM records

1. Preface

It has been more than a month since the last article was updated. The reason why I haven't been updated is that I am busy with my work recently and that I don't feel I can produce any articles that are valuable to myself and others.So during this time, the main free time is to learn technology and write GitHub, and the blog has temporarily fallen down.

The completion of this article is more like a note than a blog.Because after one year's work, I first encountered OOM problem, although the cause is relatively simple, but it is also memorable, Ha-ha.

2. Reproduction of Problems

Code source: https://github.com/jit wxs/disruptor-study/blob/master/disruptor-demo/src/test/java/jit/wxs/disruptor/demo/oom

The cause of the problem is related to Disruptor. If you don't know your classmates, you can think of it as a connected ring Queue, which is OK.

2.1 Code implementation

First create the entity class Entity that Disruptor stores. It has an object called dataList, which stores a reference to EntityData:

@Data
public class Entity {
    private long id;

    private List<EntityData> dataList;
}

@Data
@NoArgsConstructor(access = AccessLevel.PRIVATE)
@AllArgsConstructor
public class EntityData {
    private long id;

    private String message;
}

Create a Disruptor consumer, consume data from a queue, print the sequence of current consumption, and the number of objects referenced in the dataList:

class EntityEventHandler implements EventHandler<Entity> {

    @Override
    public void onEvent(Entity event, long sequence, boolean endOfBatch) throws Exception {
        // Consuming data from ringBuffer
        System.out.println("EntityEventHandler Sequence: " + sequence + ", subList size: " + event.getDataList().size());
    }
}

Create a Disruptor producer and queue data by calling the publish() method:

class EntityEventTranslator {
    private final RingBuffer<Entity> ringBuffer;

    public EntityEventTranslator(RingBuffer<Entity> ringBuffer) {
        this.ringBuffer = ringBuffer;
    }

    private static final EventTranslatorTwoArg<Entity, Long, List<EntityData>> TRANSLATOR = (event, sequence, id, dataList) -> {
        event.setId(id);
        event.setDataList(dataList);
    };

    public void publish(Long id, List<EntityData> dataList) {
        ringBuffer.publishEvent(TRANSLATOR, id, dataList);
    }
}

To create a running main class, the main business operation is to produce data in a dead loop.

public class OOMTest {
    private static int BUFFER_SIZE = 65536;

    public static void main(String[] args) throws InterruptedException {
        Disruptor<Entity> disruptor = new Disruptor<>(Entity::new, BUFFER_SIZE, DaemonThreadFactory.INSTANCE, ProducerType.SINGLE, new BlockingWaitStrategy());

        // 2. Add consumers
        disruptor.handleEventsWith(new EntityEventHandler());

        // 3. Start Disruptor
        RingBuffer<Entity> ringBuffer = disruptor.start();

        // 4. Create producers
        EntityEventTranslator producer = new EntityEventTranslator(ringBuffer);

        // 5. Dead Loop Send Events
        while (true) {
            long id = RandomUtils.nextLong(1, 100000);
            List<EntityData> dataList = mockData(RandomUtils.nextInt(10, 1000));

            producer.publish(id, dataList);

            TimeUnit.MILLISECONDS.sleep(10);
        }
    }

    private static List<EntityData> mockData(int size) {
        List<EntityData> result = Lists.newArrayListWithCapacity(size);
        for(int i = 0; i < size; i++) {
            result.add(new EntityData(RandomUtils.nextLong(100000, 1000000), RandomStringUtils.randomAlphabetic(1, 100)));
        }
        return result;
    }
}

2.2 Java VisualVM

Since JDK 1.6, the Java VisualVM has been installed with its own JVM monitoring tool. This time we use it to monitor the heap memory after the program runs.Run the program, start the Java VisualVM, and connect to the program.It can be observed that the heap size increases as the program runs.

2.3 Replicate OOM

OOM is achieved by controlling the heap size of the JVM.Configure the startup parameters to run the main class in IDEA as follows:

-Xms256m -Xmx256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=D:\disruptor_oom.hprof

The first specifies the initial size of the heap, the second specifies the maximum size of the heap, the third specifies the automatic generation of dump files when OOM occurs in the JVM, and the fourth specifies the location of the dump files.

Rerun the program, because we configured Xmx, triggering OOM when memory grows to a threshold, and generating dump files at the specified location.



3. Heap Dump

3.1 Analyzing Dump

The Emory analyzer of Eclipse is currently the most commonly used dump file analysis tool, with two versions, the plug-in version and the stand-alone version of Ecplise.Since the IDE we are using is IDEA, so Install Standalone Edition That's it.

After installation, click File -> Open Heap Dump to open the generated dump file, wait for the lower right corner to load, and select Leak Suspects Report to open the analysis report.

Since there are no interferences with this Case used, the Override page actually shows the problem as soon as the report opens:

com.lmax.disruptor.RingBuffer @ 0xf8153f78
Shallow Size: 144 B Retained Size: 232.6 MB

We pretended to be invisible and clicked Leak Suspects to view the memory leak analysis report.As shown in Figure 1, only one problem occurred in the report.Actual projects may show several, and you need to find the problem that really causes OOM.

Focus on Shallow Heap and Retained Heap fields at places 2 and 3.Professional explanations are obscure and difficult to understand, in plain words: Shallow Heap denotes the size of the object itself; Retained Heap denotes the size of the object itself and all referenced objects that can be released after its GC.

For example, in the following code, Shallow Heap represents the size of A itself, and Retained Heap represents the size of A + B (precisely, only if B is not referenced by other objects, that is, if GC releases B when releasing A, Retained Heap is the sum of both).

class A { B b; }

class B { String sss; }

Back in the report, at position 2, we show the entries array at the bottom of the RingBuffer queue with only 144 byte s of Shallow Heap and 24389304 bytes of Retained Heap, indicating that it references a large number of objects that can be GC-enabled.You can also see that each element of the entries array is displayed at 3.

3.2 Reasons for problems

The reason for the problem is that the entries array is too large, because the Entity object has a dataList of EntityData, and each Entity holds many EntityData, resulting in a larger entries array.

Although these Entity objects and the EntityData they hold can all be used by GC, unfortunately OOM has not yet waited for GC.Therefore, to solve this problem, the program must be able to support the GC.

Reduce the BUFFER_SIZE in the main class of the program from 65536 to 128, run it again, and try again.

You can see that the program memory remains stable.

BUFFER_SIZE sets the length of this entries array, which you said earlier can be interpreted as a connected ring Queue, then when the entries array is full, it is overwritten from the beginning.When overwritten, the original object loses its reference and can be GC.

IV. Conclusion

There are two general ways to generate dump files, either by using JVM parameters at program startup and automatically dump the OOM as in Section 2.3, or by dump the current program with commands:

jmap -dump:[live,]format=b,file=fileName [pid]

jmap is also a widget that is automatically installed when you install JDK to help us dump heaps.The live parameter is optional, only active objects are output to the dunmp file when selected, and the last pid specifies the process number of the java program.For example:

jmap -dump:format=b,file=/home/admin/disruptor_oom.hprof 1235

In addition, the two tools used in this paper, Java VisualVM and MooryAnalyzer, do not elaborate on their functions. I will write a special article to describe the commonly used tools in Java performance analysis.

Tags: Java jvm github JDK

Posted on Tue, 05 May 2020 01:26:07 -0400 by echoofavalon