joernio
[Bug] --import script functionality broken for scripts with statements
Describe the bug
With older joern versions it was possible to pass with --import an additional script that was run before the main --script was executed, which allowed to define certain logic or rules. Now if the script includes statements and not just function definitions the load fails with a parser error:
exception occurred while parser /tmp/2101584927108589948
An unhandled exception was thrown in the compiler.
Please file a crash report here:
https://github.com/scala/scala3/issues/new/choose
For non-enriched exceptions, compile with -Xno-enrich-error-messages.
while compiling: /tmp/2101584927108589948
during phase: parser
mode: Mode(ImplicitsEnabled)
library version: version 2.13.14
compiler version: version 3.5.2
This broke since the refactor that started with this commit:
Author: Michael Pollmeier <[email protected]> 2024-11-26 12:26:06
Committer: GitHub <[email protected]> 2024-11-26 12:26:06
Parent: 141d8f08d9844fe747edc99ae88adfaaa9fb01cb ([jssrc2cpg] Improve error handling even further (#5130))
Child: 099f4cb76fbf266d8fcd9a8f5805ec9a112f8b0c (frontends: allow multiple `--exclude` parameters (additionally) (#5133))
Branches: master, remotes/origin/andrei/remove-scala-process-builder-api, remotes/origin/dave/ruby/re-usable-script-container, remotes/origin/dave/ruby/re-usable-script-container-review, remotes/origin/markus/binSigCalc, remotes/origin/markus/libInfo, remotes/origin/master, remotes/origin/max/capturingRework, remotes/origin/michael/debug6, remotes/origin/michael/dot-export, remotes/origin/michael/upgrade
Follows: v4.0.159
Precedes: v4.0.160
upgrade scala-repl-pp (#5131)
The old description for import was import (and run) additional script(s) on startup - may be passed multiple times, while the new states: given source files will be compiled and added to classpath - this may be passed multiple times
According to the documentation (https://docs.joern.io/interpreter/#importing-additional-scripts) additional scripts can be specified.
Of course it might be possible to fully restructure the code and put everything into the script or define all logic in functions that are run by the script, but this breaks a nice feature that we are using. (Tried also to import with //> using file, but as it triggeres a pre-compile it fails with the same error.)
Never the less it is very difficult also to debug this from the scary scala error message.
To Reproduce
Here is a unit test to reproduce it:
diff --git a/joern-cli/src/test/scala/io/joern/joerncli/RunScriptTests.scala b/joern-cli/src/test/scala/io/joern/joerncli/RunScriptTests.scala
index 47e568446..b875121bb 100644
--- a/joern-cli/src/test/scala/io/joern/joerncli/RunScriptTests.scala
+++ b/joern-cli/src/test/scala/io/joern/joerncli/RunScriptTests.scala
@@ -137,6 +137,33 @@ class RunScriptTests extends AnyWordSpec with Matchers {
}
}
+ "use additional import script: --import parameter with implicit content" in new Fixture {
+ def test(scriptFile: File, outputFile: File) = {
+ val escScriptPath = outputFile.pathAsString.replace("\\", "\\\\")
+
+ scriptFile.write(s"""
+ |val fw = new java.io.FileWriter("$escScriptPath", true)
+ |fw.write("michael")
+ |fw.close()
+ """.stripMargin)
+
+ File.temporaryFile().foreach(importFile => {
+
+ importFile.write(s"""
+ |val fw = new java.io.FileWriter("$escScriptPath", true)
+ |fw.write("corleone")
+ |fw.close()
+ """.stripMargin)
+
+ ReplBridge.main(Array("--script", scriptFile.pathAsString, "--import", importFile.pathAsString))
+
+ withClue(s"$outputFile content: ") {
+ outputFile.lines.head shouldBe "michael"
+ }
+ })
+ }
+ }
+
"should return Failure if" when {
"script doesn't exist" in {
val result = ReplBridge.runScript(Config(scriptFile = Some(scriptsRoot.resolve("does-not-exist.sc"))))
Expected behavior
Would be nice to have the old and easy way of importing scripts or at least have an easy way to have multiple scripts with meaningful error messages.
Desktop (please complete the following information):
- OS: Linux Ubuntu 22.04
- Joern Version : v4.0.261,
- Java version:
openjdk 19.0.2 2023-01-17
OpenJDK Runtime Environment (build 19.0.2+7-Ubuntu-0ubuntu322.04)
OpenJDK 64-Bit Server VM (build 19.0.2+7-Ubuntu-0ubuntu322.04, mixed mode, sharing)
The semantics of importing scripts indeed changed, and --import is now only compiled, rather than interpreted. This was necessary to handle complex situations where compilation units cross-reference each other.
Your use case of 'execute this code before' is now handled by --runBefore. Sorry about not updating the docs, I'll do so once we get this resolved. But at least ./joern --help has it:
./joern --help
Usage: joern [options] [<cpg.bin>]
Script execution
--script <value> path to script file: will execute and exit
--param param1=value1 key/value pair for main function in script - may be passed multiple times
--import script1.sc given source files will be compiled and added to classpath - this may be passed multiple times
--runBefore 'import Int.MaxValue'
given code will be executed on startup - this may be passed multiple times
Note that it doesn't take entire files, but individual statements. Example usage:
$ cat foo.sc
println(foo)
$ ./joern --script foo.sc --runBefore 'val foo = 42'
executing foo.sc
42
$ ./joern --script foo.sc --runBefore 'val bar = 41' --runBefore 'val foo = bar'
executing foo.sc
41
Does this help?
add these to the script head: import root.io.joern.console.* import root.io.joern.joerncli.console.JoernConsole.* import root.io.shiftleft.codepropertygraph.cpgloading.* import root.io.shiftleft.codepropertygraph.generated.{help => _, } import root.io.shiftleft.codepropertygraph.generated.nodes.* import root.io.joern.dataflowengineoss.language.* import root.io.shiftleft.semanticcpg.language.* import io.shiftleft.codepropertygraph.Cpg import io.shiftleft.codepropertygraph.cpgloading.CpgLoader import io.shiftleft.codepropertygraph.generated.nodes. import io.shiftleft.semanticcpg.language.* import io.shiftleft.codepropertygraph.generated.nodes import io.shiftleft.codepropertygraph.generated.PropertyNames import io.joern.dataflowengineoss.language.* import scala.jdk.CollectionConverters.* implicit val resolver: ICallResolver = NoResolve implicit val finder: NodeExtensionFinder = DefaultNodeExtensionFinder
that's the default 'runbefore' code - not sure what you're trying to say.. can you elaborate?
Something like this has been bothering me for months, specifically, there are multiple scripts, using :load C :load B :load A load all the scripts in the cli, and then executing them without any problems. However, if you use joern --script A.sc --import B.sc --import C.sc this form, there is a high probability that a similar error will occur.
exception occurred while typechecking B.sc
An unhandled exception was thrown in the compiler.
Please file a crash report here:
https://github.com/scala/scala3/issues/new/choose
For non-enriched exceptions, compile with -Xno-enrich-error-messages.
while compiling: B.sc
during phase: typer
mode: Mode(ImplicitsEnabled)
...
error during script execution: Index -193 out of bounds for length 241
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: Index -193 out of bounds for length 241
at dotty.tools.dotc.util.SourceFile.lineToOffset(SourceFile.scala:161)
at dotty.tools.dotc.util.SourcePosition.lineOffsets$$anonfun$1(SourcePosition.scala:45)
at scala.runtime.java8.JFunction1$mcII$sp.apply(JFunction1$mcII$sp.scala:17)
at scala.collection.immutable.List.map(List.scala:247)
at dotty.tools.dotc.util.SourcePosition.lineOffsets(SourcePosition.scala:45)
at dotty.tools.dotc.util.SourcePosition.beforeAndAfterPoint(SourcePosition.scala:48)
at dotty.tools.dotc.reporting.MessageRendering.sourceLines(MessageRendering.scala:69)
at dotty.tools.dotc.reporting.MessageRendering.messageAndPos(MessageRendering.scala:250)
at dotty.tools.dotc.reporting.MessageRendering.messageAndPos$(MessageRendering.scala:20)
at dotty.tools.dotc.reporting.AbstractReporter.messageAndPos(AbstractReporter.scala:8)
at dotty.tools.dotc.reporting.ConsoleReporter$AbstractConsoleReporter.doReport(ConsoleReporter.scala:43)
at dotty.tools.dotc.reporting.ConsoleReporter.doReport(ConsoleReporter.scala:23)
at replpp.util.SimpleDriver$$anon$1.doReport(SimpleDriver.scala:75)
at dotty.tools.dotc.reporting.Reporter.issueUnconfigured(Reporter.scala:160)
at dotty.tools.dotc.reporting.Reporter.go$1(Reporter.scala:191)
at dotty.tools.dotc.reporting.Reporter.issueIfNotSuppressed(Reporter.scala:210)
at dotty.tools.dotc.reporting.Reporter.report(Reporter.scala:213)
at dotty.tools.dotc.report$.error(report.scala:69)
at dotty.tools.dotc.typer.ErrorReporting$.errorType(ErrorReporting.scala:32)
at dotty.tools.dotc.typer.TypeAssigner.notAMemberErrorType(TypeAssigner.scala:186)
at dotty.tools.dotc.typer.TypeAssigner.notAMemberErrorType$(TypeAssigner.scala:16)
at dotty.tools.dotc.typer.Typer.notAMemberErrorType(Typer.scala:145)
I think the reason is that in the cli, runBefore is already running, but the order of execution via --script is to compile first, then run, and runBefore is executed at runtime, but there is no default import and implicit in runBefore at compile time, resulting in the compilation already failing. To solve this problem, you need to automatically add runBefore at compile time.