OutOfMemory for WSL and Windows

[omicscoutspweber]

While starting the scan from /home/<username> On WSL running Ubuntu 20.04 an OutOfMemory occurs after starting hundreds of threads (my guess is for each recursive subdirectory) after a couple of minutes.

Running log4j-detector-2021.12.13.jar with java --version openjdk 11.0.11

        at com.mergebase.log4j.Bytes.resizeArray(Bytes.java:93)
        at com.mergebase.log4j.Bytes.streamToBytes(Bytes.java:56)
        at com.mergebase.log4j.Bytes.streamToBytes(Bytes.java:43)
        at com.mergebase.log4j.Log4JDetector.findLog4jRecursive(Log4JDetector.java:148)
        at com.mergebase.log4j.Log4JDetector.scan(Log4JDetector.java:307)
        at com.mergebase.log4j.Log4JDetector.analyze(Log4JDetector.java:332)
        at com.mergebase.log4j.Log4JDetector.analyze(Log4JDetector.java:328)
        at com.mergebase.log4j.Log4JDetector.analyze(Log4JDetector.java:328)
        at com.mergebase.log4j.Log4JDetector.analyze(Log4JDetector.java:328)
        at com.mergebase.log4j.Log4JDetector.main(Log4JDetector.java:72)```

[aveXcaesar]

Same on my Windows machine: java -jar log4j-detector-2021.12.13.jar c:\

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at com.mergebase.log4j.Bytes.resizeArray(Bytes.java:93)
        at com.mergebase.log4j.Bytes.streamToBytes(Bytes.java:56)
        at com.mergebase.log4j.Bytes.streamToBytes(Bytes.java:43)
        at com.mergebase.log4j.Log4JDetector.findLog4jRecursive(Log4JDetector.java:153)
        at com.mergebase.log4j.Log4JDetector.scan(Log4JDetector.java:312)
        at com.mergebase.log4j.Log4JDetector.analyze(Log4JDetector.java:337)
        at com.mergebase.log4j.Log4JDetector.analyze(Log4JDetector.java:333)
        at com.mergebase.log4j.Log4JDetector.analyze(Log4JDetector.java:333)
        at com.mergebase.log4j.Log4JDetector.analyze(Log4JDetector.java:333)
        at com.mergebase.log4j.Log4JDetector.analyze(Log4JDetector.java:333)
        at com.mergebase.log4j.Log4JDetector.analyze(Log4JDetector.java:333)
        at com.mergebase.log4j.Log4JDetector.analyze(Log4JDetector.java:333)
        at com.mergebase.log4j.Log4JDetector.analyze(Log4JDetector.java:333)
        at com.mergebase.log4j.Log4JDetector.analyze(Log4JDetector.java:333)
        at com.mergebase.log4j.Log4JDetector.analyze(Log4JDetector.java:333)
        at com.mergebase.log4j.Log4JDetector.main(Log4JDetector.java:72)

[jangatzke]

Did you try to give java more memory by adding -Xmx1024m as command line parameter?

[drnimrod]

Use this to avoid out of memory errors: java -jar -Xmx1024m log4j-detector-2021.12.13.jar --verbose

[aveXcaesar]

No! I will try it... Thx!

[aveXcaesar]

Sorry, same problem. It needs some more time to crash now. I used the following call: java -jar -Xmx1024m log4j-detector-2021.12.13.jar c:\

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at com.mergebase.log4j.Bytes.resizeArray(Bytes.java:93)
        at com.mergebase.log4j.Bytes.streamToBytes(Bytes.java:56)
        at com.mergebase.log4j.Bytes.streamToBytes(Bytes.java:43)
        at com.mergebase.log4j.Log4JDetector.findLog4jRecursive(Log4JDetector.java:153)
        at com.mergebase.log4j.Log4JDetector.scan(Log4JDetector.java:312)
        at com.mergebase.log4j.Log4JDetector.analyze(Log4JDetector.java:337)
        at com.mergebase.log4j.Log4JDetector.analyze(Log4JDetector.java:333)
        at com.mergebase.log4j.Log4JDetector.analyze(Log4JDetector.java:333)
        at com.mergebase.log4j.Log4JDetector.analyze(Log4JDetector.java:333)
        at com.mergebase.log4j.Log4JDetector.analyze(Log4JDetector.java:333)
        at com.mergebase.log4j.Log4JDetector.analyze(Log4JDetector.java:333)
        at com.mergebase.log4j.Log4JDetector.analyze(Log4JDetector.java:333)
        at com.mergebase.log4j.Log4JDetector.analyze(Log4JDetector.java:333)
        at com.mergebase.log4j.Log4JDetector.analyze(Log4JDetector.java:333)
        at com.mergebase.log4j.Log4JDetector.analyze(Log4JDetector.java:333)
        at com.mergebase.log4j.Log4JDetector.main(Log4JDetector.java:72)

I think/hope the option --verbose has no impact to the problem.

[omicscoutspweber]

I limited the available memory to 1GB using the command java -jar -Xmx1024m log4j-detector-2021.12.13.jar --verbose /home/\<username\> . I can can confirm @aveXcaesar 's observation, that it now fails more quickly. The --verbose flag made no difference.

The failure happened while traversing a particular file that had more than 100MB in size.

[aveXcaesar]

That make sense. With ´--verbose´ I can see that the last logged file as a size of 133MB.

[phbreitbach]

I added the Pull Request above to further improve OutOfMemory Handling ... Please fell free to share feedback :-)