spring-projects
parsing using open-rewrite is dramatically slow for larger source files
I am testing with a project that contains some large files (160k lines of java source). The overall JDT parsing seems to be okay for those files, but the open rewrite based parsing takes ages for those files and keeps the CPU busy (at least one core) for a very long time.
Looking at this briefly shows threads running with this:
ForkJoinPool.commonPool-worker-10 Runnable CPU usage on sample: 1s 1ms
java.util.regex.Pattern$Start.match(Pattern.java:3608)
java.util.regex.Matcher.search(Matcher.java:1728)
java.util.regex.Matcher.find(Matcher.java:772)
org.openrewrite.java.isolated.ReloadableJava17ParserVisitor.lambda$visitVariables$28(ReloadableJava17ParserVisitor.java:1387)
org.openrewrite.java.isolated.ReloadableJava17ParserVisitor$$Lambda$1028.0x000000080123ec18.get()
org.openrewrite.java.isolated.ReloadableJava17ParserVisitor.visitVariables(ReloadableJava17ParserVisitor.java:1394)
org.openrewrite.java.isolated.ReloadableJava17ParserVisitor.visitVariable(ReloadableJava17ParserVisitor.java:1342)
org.openrewrite.java.isolated.ReloadableJava17ParserVisitor.visitVariable(ReloadableJava17ParserVisitor.java:71)
com.sun.tools.javac.tree.JCTree$JCVariableDecl.accept(JCTree.java:1045)
com.sun.source.util.TreePathScanner.scan(TreePathScanner.java:86)
org.openrewrite.java.isolated.ReloadableJava17ParserVisitor.convert(ReloadableJava17ParserVisitor.java:1478)
org.openrewrite.java.isolated.ReloadableJava17ParserVisitor.convert(ReloadableJava17ParserVisitor.java:1507)
org.openrewrite.java.isolated.ReloadableJava17ParserVisitor.convertAll(ReloadableJava17ParserVisitor.java:1544)
org.openrewrite.java.isolated.ReloadableJava17ParserVisitor.visitMethod(ReloadableJava17ParserVisitor.java:950)
org.openrewrite.java.isolated.ReloadableJava17ParserVisitor.visitMethod(ReloadableJava17ParserVisitor.java:71)
com.sun.tools.javac.tree.JCTree$JCMethodDecl.accept(JCTree.java:953)
com.sun.source.util.TreePathScanner.scan(TreePathScanner.java:86)
org.openrewrite.java.isolated.ReloadableJava17ParserVisitor.convert(ReloadableJava17ParserVisitor.java:1478)
org.openrewrite.java.isolated.ReloadableJava17ParserVisitor.convert(ReloadableJava17ParserVisitor.java:1507)
org.openrewrite.java.isolated.ReloadableJava17ParserVisitor.convertStatements(ReloadableJava17ParserVisitor.java:1616)
org.openrewrite.java.isolated.ReloadableJava17ParserVisitor.convertStatements(ReloadableJava17ParserVisitor.java:1598)
org.openrewrite.java.isolated.ReloadableJava17ParserVisitor.visitClass(ReloadableJava17ParserVisitor.java:472)
org.openrewrite.java.isolated.ReloadableJava17ParserVisitor.visitClass(ReloadableJava17ParserVisitor.java:71)
com.sun.tools.javac.tree.JCTree$JCClassDecl.accept(JCTree.java:860)
com.sun.source.util.TreePathScanner.scan(TreePathScanner.java:86)
org.openrewrite.java.isolated.ReloadableJava17ParserVisitor.convert(ReloadableJava17ParserVisitor.java:1478)
org.openrewrite.java.isolated.ReloadableJava17ParserVisitor.convert(ReloadableJava17ParserVisitor.java:1507)
org.openrewrite.java.isolated.ReloadableJava17ParserVisitor.convertStatements(ReloadableJava17ParserVisitor.java:1616)
org.openrewrite.java.isolated.ReloadableJava17ParserVisitor.convertStatements(ReloadableJava17ParserVisitor.java:1598)
org.openrewrite.java.isolated.ReloadableJava17ParserVisitor.visitClass(ReloadableJava17ParserVisitor.java:472)
org.openrewrite.java.isolated.ReloadableJava17ParserVisitor.visitClass(ReloadableJava17ParserVisitor.java:71)
com.sun.tools.javac.tree.JCTree$JCClassDecl.accept(JCTree.java:860)
com.sun.source.util.TreePathScanner.scan(TreePathScanner.java:86)
org.openrewrite.java.isolated.ReloadableJava17ParserVisitor.convert(ReloadableJava17ParserVisitor.java:1478)
org.openrewrite.java.isolated.ReloadableJava17ParserVisitor.convertAll(ReloadableJava17ParserVisitor.java:1531)
org.openrewrite.java.isolated.ReloadableJava17ParserVisitor.visitCompilationUnit(ReloadableJava17ParserVisitor.java:518)
org.openrewrite.java.isolated.ReloadableJava17ParserVisitor.visitCompilationUnit(ReloadableJava17ParserVisitor.java:71)
com.sun.tools.javac.tree.JCTree$JCCompilationUnit.accept(JCTree.java:614)
com.sun.source.util.TreePathScanner.scan(TreePathScanner.java:86)
org.openrewrite.java.isolated.ReloadableJava17Parser.lambda$parseInputs$0(ReloadableJava17Parser.java:177)
org.openrewrite.java.isolated.ReloadableJava17Parser$$Lambda$906.0x00000008011ef328.apply()
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
java.util.Iterator.forEachRemaining(Iterator.java:133)
java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845)
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
org.openrewrite.java.isolated.ReloadableJava17Parser.parseInputs(ReloadableJava17Parser.java:199)
org.openrewrite.java.Java17Parser.parseInputs(Java17Parser.java:43)
org.springframework.ide.vscode.commons.rewrite.java.ORAstUtils.parseInputs(ORAstUtils.java:233)
org.springframework.ide.vscode.boot.java.rewrite.RewriteCompilationUnitCache.doParse(RewriteCompilationUnitCache.java:277)
org.springframework.ide.vscode.boot.java.rewrite.RewriteCompilationUnitCache.lambda$7(RewriteCompilationUnitCache.java:234)
org.springframework.ide.vscode.boot.java.rewrite.RewriteCompilationUnitCache$$Lambda$632.0x00000008010a4350.get()
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1760)
java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
Ouch... any user-facing side effects of slow parsing? If there is anything then perhaps it is best to switch back to JDT to detect.mark problems and leave quick fixes and assists to rewrite... The stack trace suggest forwarding this issue to rewrite in my opinion as this is just plain parsing without any extras I'm doing after the source is parsed.
Next test in this area: project reconciling with a small project containing large source files goes crazy, quickly running out of heap space on the language server side, CPU spiking all the time.
Thread dump looks like this is the important piece: thread-dump-project-reconciling-large-sources.txt
The extremely deep call stack doesn't look good to me...
@martinlippert seems like this is another one for Rewrite folks to look at... What is the project you're trying? Still the antlr parser?
@BoykoAlex Yes, and I already posted a comment to the open-rewrite issue that you raised... :-)