maven-mvnd
maven-mvnd copied to clipboard
apache
Too many open files on Mac OS with JDK 11 and mvnd 0.8.2
We have a very large project and in our day to day development also when we upgraded from JDK8 to JDK11, when running regular maven builds we used to get Too many open files error. We circumvented this in 2 ways.
- On mac , we created a max.limitfiles.plist file with a huge 10 million value.
- We passed
-XX:-MaxFDLimitunder.mvn/jvm.config
Together after these 2 running mvnw clean install -DskipTests -T16 worked fine as opposed to giving Too many open files.
The system is Apple M1 Max with JDK 11 and maven 3.8
We wanted to try maven daemon and we faced the same Too many open files issue when building our project. Doing lsof shows that it happens around 10k files ( since that is the JVM limit on Mac OS ) which is why -XX:-MaxFDLimit was needed.
We tried to create .mvn/mvnd.properties and put mvnd.jvmArgs=-XX:-MaxFDLimit however we still get this error.
On the shell we see that ulimit -a is set to the 10 million file value.
We can also see in the command that runs the -Dmvnd.jvmArgs having the above value.
Not sure if there is any steps that I could use to debug this issue.
Running ps on the maven daemon process
/nix/store/5ichab6s1ijq5im4vly9lkcsm8m8a8ys-jdk11-11.0.11.3/bin/java -classpath /nix/store/2mb6clvrn68kwq90v2qq3l1ilyaq51yg-mvnd/mvn/lib/ext/mvnd-common-0.8.2.jar:/nix/store/2mb6clvrn68kwq90v2qq3l1ilyaq51yg-mvnd/mvn/lib/ext/mvnd-agent-0.8.2.jar -javaagent:/nix/store/2mb6clvrn68kwq90v2qq3l1ilyaq51yg-mvnd/mvn/lib/ext/mvnd-agent-0.8.2.jar -XX:-MaxFDLimit -Dmvnd.home=/nix/store/2mb6clvrn68kwq90v2qq3l1ilyaq51yg-mvnd -Dmaven.home=/nix/store/2mb6clvrn68kwq90v2qq3l1ilyaq51yg-mvnd/mvn -Dmaven.conf=/nix/store/2mb6clvrn68kwq90v2qq3l1ilyaq51yg-mvnd/mvn/conf -Dmvnd.java.home=/nix/store/5ichab6s1ijq5im4vly9lkcsm8m8a8ys-jdk11-11.0.11.3 -Dlogback.configurationFile=/nix/store/2mb6clvrn68kwq90v2qq3l1ilyaq51yg-mvnd/conf/logback.xml -Dmvnd.id=dc67aa8c -Dmvnd.daemonStorage=/Users/mkodnani/.m2/mvnd/registry/0.8.2 -Dmvnd.registry=/Users/mkodnani/.m2/mvnd/registry/0.8.2/registry.bin -Dmvnd.socketFamily=inet -Dmvnd.home=/nix/store/2mb6clvrn68kwq90v2qq3l1ilyaq51yg-mvnd -Djdk.java.options=--add-opens java.base/java.io=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/sun.net.www.protocol.jar=ALL-UNNAMED --add-opens java.base/sun.nio.fs=ALL-UNNAMED -Dmvnd.noDaemon=false -Dmvnd.debug=false -Dmvnd.idleTimeout=3h -Dmvnd.keepAlive=100ms -Dmvnd.extClasspath= -Dmvnd.coreExtensions=com.zeus:zeus-extension:1.0.9 -Dmvnd.jvmArgs=-XX:-MaxFDLimit -Dmvnd.enableAssertions=false -Dmvnd.expirationCheckDelay=10s -Dmvnd.duplicateDaemonGracePeriod=10s -Dmvnd.socketFamily=inet org.mvndaemon.mvnd.common.MavenDaemon
you can see that -Dmvnd.jvmArgs=-XX:-MaxFDLimit is read from .mvn/mvnd.properties file.
Version:
❯ mvnd -version
mvnd 0.8.2 darwin-aarch64 native client (2bba2d6a4d3a5012ddf9f1f42a22784cad4011e3)
Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)
Java version: 11.0.11, vendor: Azul Systems, Inc., runtime: /nix/store/5ichab6s1ijq5im4vly9lkcsm8m8a8ys-jdk11-11.0.11.3
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "11.5", arch: "x86_64", family: "mac"
Ulimit on the shell:
ulimit -a
Maximum size of core files created (kB, -c) 0
Maximum size of a process’s data segment (kB, -d) unlimited
Maximum size of files created by the shell (kB, -f) unlimited
Maximum size that may be locked into memory (kB, -l) unlimited
Maximum resident set size (kB, -m) unlimited
Maximum number of open file descriptors (-n) 10000000
Maximum stack size (kB, -s) 8176
Maximum amount of CPU time in seconds (seconds, -t) unlimited
Maximum number of processes available to current user (-u) 16000
Maximum amount of virtual memory available to each process (kB, -v) unlimited
> launchctl limit maxfiles
maxfiles 10000000 10000000
I ran into a similar issue when I tried to generate sources with a java program that loads a lot of jars itself. I started it in an own class world. After the goal was finished, my custom plugin was not able to unload the classloader. So after a couple of goal invocations, the maxfiles limit was reached. When the maven process was shut down they were released.
hello @ppalaga ! thank you for looking into this.
it is -XX:+MaxFDLimit with -.
have a look at https://gist.github.com/ndimiduk/a6c2aa781c20fb8bb9c20abbcf5bac4f
I see two things to investigate:
- use the
mvnd.pluginRealmEvictPatternto get rid of plugins that do not unload properly - improve the expiration checks to check if the number of open fd is nearing the max amount (if those information can be retrieved)
I see two things to investigate:
- use the
mvnd.pluginRealmEvictPatternto get rid of plugins that do not unload properly- improve the expiration checks to check if the number of open fd is nearing the max amount (if those information can be retrieved)
@gnodet thank you for looking into this.
please correct me if I am wrong, at the moment your suggestions ask to optimize the resources and stay below the 10k file limit. but what about allowing the flag -XX:-MaxFDLimit to reach the jvm?
thank you!
I see two things to investigate:
- use the
mvnd.pluginRealmEvictPatternto get rid of plugins that do not unload properly- improve the expiration checks to check if the number of open fd is nearing the max amount (if those information can be retrieved)
@gnodet thank you for looking into this. please correct me if I am wrong, at the moment your suggestions ask to optimize the resources and stay below the 10k file limit. but what about allowing the flag
-XX:-MaxFDLimitto reach the jvm? thank you!
Afaik, the JVM args problem has been fixed in https://github.com/apache/maven-mvnd/pull/751
hello @gnodet ! do you know if the fix is available also in the version 0.9.0? thank you! I tried it out on 0.9.0 and I am getting the usual "Too many open files"
The fix for the JVM args has been back ported with https://github.com/apache/maven-mvnd/commit/f2117ccfffac53141f80bca40c839e6a6e3b9bcc
Have you added the -XX:-MaxFDLimit flag ?
If it does not work for you, please reopen a bug and provide a test case / reproducer.
thank you @gnodet ! It does not work or I was not able to make it work. last try it was with version 1.0-m6
hey @gnodet !
I am not able to replicate the problem with the following command mvnd -Dmvnd.jvmArgs=-XX:-MaxFDLimit clean install and the following configuration
- mvnd 1.0-m8
- mac os x 14.4.1
maybe this can be rejected/closed now.
thank you very much!
I've spent far too much time in the last day looking into this. I think I know what is happening though.
Graalvm doesn't support the MaxFDLimit flag at least according to this across both jdk/native variants in all versions:
https://chriswhocodes.com/graalvm_native_image_jdk17_options.html
Looking at the source code for graalvm: https://github.com/oracle/graal/
There isn't any real option for interacting with RLIMIT_NOFILE except in a weird corner case during PosixNativeLibrary initialization.
Hey @telemenar in my case the problem is intermittent, so yes I think it has some limitation, not sure how/why or if it is really because of GraalVM/native-image handling of MaxFDLimit. Could you offer some light here @fniephaus @alina-yur ?
I would rule out GraalVM in this case. It's only the client which is compiled to native, and that one mostly communicate with the daemon, so I don't really see a reason why it would reach a limit, especially as it does after each build.