maven-mvnd
maven-mvnd copied to clipboard
apache
javad?
Hi,
Wonder if you investigated some wider "daemon" scope for mvnd. I'm coming from surefire which is super slow when you have a lot of modules and a few tests (basically creating the jvm is way slower than running the tests).
Idea would be to keep a JVM up and running - potentially the same as mvnd one? - and reuse it for surefire executions.
I know it requires some investment and patching of surefire (but hey, we have a javaagent now, no? ;)) but I think it can be worth.
Hope it makes sense.
Romain
Isn't this a matter of Surefire alone? I do not see how mvndaemon could help here. Surefire could perhaps do some classloader juggling much like Quarkus dev mode.
Maven ecosystem is not related to daemon, mvnd is. Therefore, reusing a daemon is a mvnd concern. Here mvnd would replace surefire by daemon-friendly-surefire-maven-plugin in the pom which would use this mecanism (a bit like mvnd client/server but injected into the build with a 3rd party plugin). An alternative is to replace surefire java command by the daemon one which would also enable to leverage this feature. Quarkus dev mode do not solve that at all since the goal there is to be able to share a hot JVM between modules (why I spoke of daemon) and "potentially" reuse existing classloader (I assume all deps but target/classes, target/test/classes) which would make it fast. Replacing the JVM creation by a socket communication can really make it faster since most of the time in tests in a multimodule project is generally classloading when modules are simple.
Maven ecosystem is not related to daemon, mvnd is. Therefore, reusing a daemon is a mvnd concern.
I thought that mvndaemon going under Maven might mean that plugins will, in cases where it makes sense, become more daemon-aware, i.e. that they better adjust to the fact that their own code may be cached over multiple executions.
Here mvnd would replace surefire by daemon-friendly-surefire-maven-plugin in the pom which would use this mecanism (a bit like mvnd client/server but injected into the build with a 3rd party plugin). An alternative is to replace surefire java command by the daemon one which would also enable to leverage this feature.
Sounds like a duplication of surefire with a lot of potential for incompatibilities. The speed gains may still be worth of doing it.
Quarkus dev mode do not solve that at all since the goal there is to be able to share a hot JVM between modules (why I spoke of daemon) and "potentially" reuse existing classloader (I assume all deps but target/classes, target/test/classes) which would make it fast.
Quarkus dev mode does exactly that. It is basically a maven mojo running until the user presses CTRL+C, watching for changes in the source tree and re-compiling classes, re-deploying the user app and re-running tests if necessary. It tries to re-use as many class loaders as possible, dropping only the changed ones. I guess surefire could be made to work in a similar manner.
I thought that mvndaemon going under Maven might mean that plugins will, in cases where it makes sense, become more daemon-aware, i.e. that they better adjust to the fact that their own code may be cached over multiple executions.
Guess it is something to discuss on the list but I see it as being an integration on top of maven way saner in time.
Sounds like a duplication of surefire with a lot of potential for incompatibilities. The speed gains may still be worth of doing it.
Maybe a dozen of lines since at the end it will use surefire itself ;). It already have this kind pluggability AFAIK and it is really about changing the exec done to use a surefire-d at the end.
Quarkus dev mode does exactly that.
You mean does not that? quarkus is about a single module like meecrowave or tomee plugins reloader so it does not reach the 2 goals of that issue:
- reuse accross executions
- (if enabled) reuse accross modules
2 can be done at surefire level but with 1 the boost is way higher so 1 makes more sense than 2 to me so I think 1 should be tried first probably.
Hope it makes sense
My understanding of surefire internals is rather foggy. I assume it spawns a new JVM on each mojo execution and I thought this could be replaced by a set of class loaders, some of those cached in whatever environment surefire is runnung in. If it is in a stock Maven, the caching would be effective across multiple modules in the same Maven process. And if it is inside mvndaemon, the cache could be effective across multiple mvnd client invocations. I thought the way the class loader cache is accessed can be the same in both cases and I thought it might be a daemon agnostic way. But maybe I am fully wrong assuming that surefire's child JVMs can be replaced by class loaders. If they cannot, maybe the idea of a daemon JVM makes sense.
There is a "dont fork" option but it never works and is not reuse oriented since surefire is not itself. Im not sure i see how mvnd would enable that to work without being mvnd aware? For me it is a mvnd feature, kind of mvnd mvn extension. Other impl sound overcomplicated to me.
The concept of JVM daemon sounds like an application server, doesn't it? If you take WildFly and remove all modules, you get a JVM daemon :) Then you can use Arquillian to reuse WildFly as a test runner.
Arquillian could perhaps be used also for JVM, if you implement some kind "Arquillian container" that will keep it running and manage the deploying and undeploying of the tested part.
There are several other such solutions that could be used to run tests. For instance, JSR batch could be used to re-run the same tests repeatedly with different input.
Well arquillian is dead-ish and it looks a bit overkill to me since there it is 100% about caching jars/folders in classloader but they are on the filesystem (and when they are no more - like surefirebooter - you drop the cache). Similarly, JBatch is not designed for that but to run tasks with tracking - here we don't care about that last part, do we?
so think a specific caching like the ones in mvnd are saner since they will be more efficient, bring less dependencies issues and management in time and will likely provide a lower contribution step.
To rephrase my answer: I don't think we should think of it in terms of "we should bring this lib" but more in terms of features since technically there is nothing really critical behind the impl IMHO. Guess the most time consuming will be to measure what are the actual gains of each scenario (hot JVM with no classloader cache, hot JVM with classloader cache etc...).
Hope it makes sense.