djl
djl copied to clipboard
deepjavalibrary
Shape safe API in Scala: verify tensor compatibility at compile-time
Description
We would like to introduce a scala API that actively validates the feasibility of tensor operations while being written, NOT while being tested or deployed. The API will be based on the scala library shapesafe, which I'm co-maintaining.
The capability of shapesafe is a superset of several prior arts (see references): It validates not only tensor with literal shape, but also tensor as an output of heavy computations (e.g. einsum or several NN layers). This makes both prototyping & critical production less error-prone, while avoiding computation overhead after deployment (as type information are erased at runtime)
After integration, a typed tensor API will be the main entry point to define an NN architecture. Calling each of its method will define both compile-time shape and runtime computation at the same time, yielding more typed tensors.
Unfortunately this API will only be for Scala, maybe it is not a bad choice considering the overlapping audience of Apache Spark.
Will this change the current api? How?
No, but a new API exclusive to Scala user can be introduced which enables this feature
Who will benefit from this enhancement?
ML Users of Scala, Apache Spark, CHISEL, SpinalHDL ... who are prone to make mistakes caused by mismatched tensor sizes.
References
Prior Art
- Evan Spark for first showing the possibility
- Tongfei Chen et al. for Nexus
- Breandan Considine et al. for Kotlin∇
- Dougal Maclaurin et al. for Dex
- Maxime Kjaer et al. for tf-dotty
Hi @tribbloid. This seems pretty interesting.
Is this something you are planning to work on?
Yeah, actually I already made a promise to one of your colleagues, @lanking520 (is he still in the team?), to start the integration last year. But then the other committer dropped out for his PhD, so I have to do it on my own, and much slower.
Who is the specialist for Scala/Spark integration?
@tribbloid please let me know if you have any Spark/Scala related issues
@lanking520 Nice! I thought you moved to another team.
At this moment, there is no scala code in master branch, should the new API be in a new module?
@lanking520 Nice! I thought you moved to another team.
At this moment, there is no scala code in master branch, should the new API be in a new module?
do you think it's better have another repo? Since we have our deepjavalibrary org, we can create a new repo for scala.
I see no problem, my code is already compiled using Gradle (same as DJL core), but I also think it should be your decision (or the lead Scala committer of DJL, hopefully there is one)