KEEP
KEEP copied to clipboard
Kotlin
Kotlin Scripting Support
https://github.com/Kotlin/KEEP/blob/master/proposals/scripting-support.md
replaces https://github.com/Kotlin/KEEP/blob/master/proposals/script-definition-template.md
This looks really promising!
Would this allow me to distribute a single self contained script template which could then be opened in Intellij without any other configuration being required? Ideally I could distribute a script template with a reference to the script definition and Intellij would automatically download this definition from Maven and provide rich code completion.
@DaanDeMeyer this is not yet in this KEEP, but it could be an interesting addition. I'll think about it. Thanks!
@ligee 2 questions:
- How is Spek related to scripting?
- What about unit testing for Kotlin Scripts? (tests for scripts themselves)?
Thanks!
Great that you guys are planning to push scripting to a new level. It's an amazing KEEP and I especially liked the bits about declarative bootstrapping of an IDE dependency and runtime environment. It's also great to read how you envision to cover the whole range from build-files to shell scripts with a unified framework addition.
Though, it reads like a very ambitious plan to me including lots of sub-projects. So I'd be curious about the indented time-frame.
-
You mentioned the difficulties of parameterized shebangs, and indeed they don't work across shells, so using a declarative approach is the way to go I think.
-
It would be helpful to spec out the declarations first, so that third party tools like my kscript (see https://github.com/holgerbrandl/kscript) could support these declarations rather soon in order to fill the gap until an official JB solutions is ready. Once the latter is done, kscript would be obsolete for good.
Looking forward to kontlinconf so that we can chat more about it!
@artem-zinnatullin: About Spec - it is described here - https://github.com/Kotlin/KEEP/blob/scripting/proposals/scripting-support.md#script-based-dsl. Basically, it means using scripting infrastructure to produce regular project class files from script-like sources with reduced boilerplate, like necessity to place spec tests inside a class. About unit testing scripts - it is, of course, possible to use script runner inside tests, but we do not plan any additional helpers/facilities to simplify scripts testing. Maybe we'll consider it later though.
@holgerbrandl, thank you for the feedback. We will try to keep the size of the work here under control. Significant parts of the infrastructure changes described here are, in fact, reshapings and unifyings of the existing scripting infrastructure. But there should be, of course, some particular implementations on top of that, and implementing them will take time, but with clearly defined interfaces we may be able to outsource it partly to the community. But at the moment it is work in progress, so the plans could change. As of the specs for declarations - we'll try to keep it mind. But first, we have to finish more general infrastructure specs.
@ligee
About Spec - it is described here - https://github.com/Kotlin/KEEP/blob/scripting/proposals/scripting-support.md#script-based-dsl
Yep, I saw Spek there, that's why I asked :)
like necessity to place spec tests inside a class
Spek is a pretty complex test framework that requires test engine/runner to work (currently Spek uses JUnit5 infrastructure). Also, usually you run tests through build system, not sure how scripting will handle that.
This can save one nesting level by removing top-level class (which doesn't look as a pain point atm), but on the other hand running tests seems to become much more complicated. Technically everything is possible, but I don't think this is direction we want to move Spek into at the moment.
cc @raniejade, @hhariri PTAL
@ligee for script support, it looks like you have something roughed in for whitelist/blacklisting of functions/classes that can be accessed. This will be important for us to embed Kotlin scripting in Elasticsearch or other engines like Arango, Neo4j, etc. Ensuring what can be called at any given moment, or even what classes can be loaded. Lastly, time limiting of the script running, or loop iteration limiting protections as well (endless loops kill these type of systems). The more integrated this protection can be, the better.
I'm working now on the updated proposal, along with the series of the prototypes. As soon as it will be ready, I'll publish the updated version. Hopefully, it will happen within a few weeks from now.
@ilya-g please add the requirement: Between compilation and class loading, there should be the option for a verifier that can approve or reject a script. This allows secure scripting to be tied into anything that executes those steps concurrently without needing to re-write every permutation of script runners that might be created. Allowing something like Cuerentena integration https://github.com/kohesive/cuarentena ... There should never be a hard-coded "compile and load class" without the ability to interject between those steps, before class loading.
@apatrida Thanks. In cases where the script content came from outside of development, e.g. some simple formula for calculating P&L etc. It is critical to validate before loading for the feature to be usable in production.
The 1.2.50 is out with the new experimental scripting support. The KEEP is updated too, reflects now the approach taken with 1.2.50 and actual implementation status. Not all feedback is incorporated (yet) into the KEEP, but I believe that what we have now is a good foundation that could be developed in the right direction relatively quick and easy. Any feedback will be highly appreciated!
@apatrida, the current approach allows to write a scripting host that will get a compiled script before instantiating/evaluating it and can perform any checks at that point. Could it be an acceptable solution for your case, or you rather believe that we should introduce another entity - a Checker - with a predefined interface, that could be used with standard hosts?
@ligee it should be after bytecode is generated but before any class is loaded to prevent attacks during static initialization. And would be nice if many implementations that might exist of scripting hosts don't all need rewritten to add verification. So some plugin point that can be exposed in the common hosts and hopefully whatever wrapping people do around those.
I'll look at what is in 1.2.50 and see how it is currently set up.
@apatrida it could be done now exactly at the moment you described, but having an extension point for it in the standard host interface looks like a good idea, I'll think about it.
@ligee As noted by @rgoldberg in #124, kotlin-scripting-common is listed twice in the "Implementation status" section, which is probably a typo
It would be great if this optionally supported continuations via an implicit surrounding runBlocking call. Avoid nasty boilerplate and allow suspending functions to be called from top level (without nesting).
@LouisCAD to clarify, I really meant calling them at the toplevel e.g. without runBlocking here
https://github.com/yschimke/oksocial/blob/master/src/test/kotlin/commands/text-to-speech.kts#L71-L78
I've been playing around with this and noticed that the cache interface for the compiled scripts doesn't seem to accept a script source reference when storing the compiled scripts. I'm curious then how the get() is able to return an instance based on a script source?
@cmorriss, seems a bug, thanks for pointing out! It was lost during the last refactoring when the source was removed from the configuration.
I haven't found any good examples of how to actually setup a script project while having full IDE support and make it run. 😢
I've noticed that this seems to require access to the full kotlin compiler. Our use case is to run it as an embedded component, allowing scripts to be incorporated as sandboxed plugins in a backend service. It would be great if this could run utilizing kotlin-compiler-embedded as a dependency. It pulls in fewer dependencies and would allow this to more easily run as a library.
Just to add more context, when I try to run using the embedded compiler, I get the following exception:
java.lang.NoClassDefFoundError: com/intellij/openapi/util/Disposer
at kotlin.script.experimental.jvmhost.impl.KJVMCompilerImpl.compile(KJVMCompilerImpl.kt:99)
at kotlin.script.experimental.jvm.JvmScriptCompiler.compile$suspendImpl(jvmScriptCompilation.kt:43)
at kotlin.script.experimental.jvm.JvmScriptCompiler.compile(jvmScriptCompilation.kt)
at com.amazon.fuselets.kotlinscript.FuseletScriptCompiler.compile(FuseletScriptCompiler.kt:19)
at com.amazon.fuselets.kotlinscript.FuseletScriptingHost$eval$1.doResume(FuseletScriptingHost.kt:28)
at com.amazon.fuselets.kotlinscript.FuseletScriptingHost$eval$1.invoke(FuseletScriptingHost.kt)
at com.amazon.fuselets.kotlinscript.FuseletScriptingHost$eval$1.invoke(FuseletScriptingHost.kt:14)
at kotlin.script.experimental.host.BasicScriptingHost$runInCoroutineContext$1.doResume(BasicScriptingHost.kt:30)
It looks like this class, among others, has a modified package name (I'm guessing to prevent collisions when running as an embedded component) of org.jetbrains.kotlin.com.intellij.openapi.util instead of the normal com.intellij.openapi.util. Maybe these classes can be encapsulated and have the system catch the first ClassNotFoundException and pick the right one under the covers for future calls and instance creation needs. Just a thought.
@cmorriss In fact, there is no difference in size and dependencies between "full" compiler and embeddable one. The latter is intended for usages in the environments already containing some of the libs packed into the jar, e.g. IDEA's openapi. And it also requires the other components to be remapped similarly, that's why you're getting the error above. We do not provide yet the appropriate scripting components for using with the embeddable compiler (and I'm not sure we want to do it, taking into account related problems and misunderstandings). In short, I do not recommend to do it. Please use the "full" compiler instead.
@ligee Thanks for the response. While it is possible to use the full compiler, we'll have to figure out a way to do so in a contained classloader as our particular use case does fall under the "environments already containing some of the libs packed into the jar" scenario.
In particular, this goes beyond just things like Google Guava. It actually includes the Kotlin standard libraries themselves. Our use case is to utilize Kotlin scripting as a backend service plugin language running in a tightly contained sandbox. This allows the logic of our services to be tweaked through these scripts by providing functional business logic in well defined extension points.
These services all depend on Kotlin at a 1.2.x or 1.3.x level instead of a specific version, like 1.2.60. This allows for backwards compatible updates to be deployed without the need to update a ton of version numbers throughout our infrastructure. However, the compiler is clearly tied to a very specific version of Kotlin.
The problem is that by including the full compiler as part of our plugin library, it can cause conflicts as the Kotlin libraries that the service depends on may diverge from the version that is included in the compiler since they can be deployed and updated independently.
The only way we can avoid this issue currently would be to load the compiler in a separate classloader, perform the compilation there, and then run the classes in the classloader of the primary service. This can work (I hope!) as we can at least be sure that the Kotlin compiler is at or below the version of the Kotlin standard library for the service. We just can't be sure they're at exactly the same version level.
I believe that running with the embeddable version of the Kotlin compiler would solve these issues as it's embedded libraries would not conflict with those of the service embedding it.
In the mean time I'll work on the classloading solution, but would greatly appreciate if using the embeddable version of the compiler could be investigated further. I'd wager we're not the only use case for this, but I could be wrong.
Would be happy to hear I'm wrong about this as well!!!!
@cmorriss,@ligee, Hello, is anyone can tell me how can I define my kotlin script that has some functions with parameters, and how can I visit that functions in my custom project? With the new experimental scripting support. Thanks!!
