kotlin-spark-api
kotlin-spark-api copied to clipboard
Kotlin
This projects gives Kotlin bindings and several extensions for Apache Spark. We are looking to have this as a part of Apache Spark 3.x
While `Dataset` works, whereever a schema needs to be created, such as `Dataset
https://github.com/prokod/gradle-crossbuild-scala/pull/117
Is it possible to use that instead of rewriting parts of the scala code?
I was trying to add the following line in my build.gradle file as said [here](https://github.com/Kotlin/kotlin-spark-api/wiki/Quick-Start-Guide#building-the-application-with-gradle): ``` implementation 'org.jetbrains.kotlinx.spark:kotlin-spark-api-3.2_2.12:1.1.0' // Apache Spark ``` The build.gradle file looks as follows: ``` plugins...
As discussed in the issue https://github.com/JetBrains/kotlin-spark-api/issues/100 it would be nice to have more stdlib functions to work with Datasets too, since it's one of Kotlin's selling points. I've started converting...
Currently when calling `copy(_x = ...)` on a Tuple, the new values need to have the same type as the original Tuple. This might however not be what the user...
https://github.com/Kotlin/KEEP/blob/master/proposals/context-receivers.md Context receivers will allow functions like ```kotlin @context(SparkSession) inline fun List.toDS() = toDS(spark) ``` to function without needing a separate `KSparkSession`. It could maybe also help in other places!...
One of the things that makes Kotlin so great to work with, compared to other languages, is the extensive and declarative standard library functions. Functions like `mapNotNull { }` and...
There seems to be no way to define data classes where the data class encoder produces a Spark schema with fields of type `Decimal(38, 0)`. The natural approach would be...
3.4+
Fixes https://github.com/Kotlin/kotlin-spark-api/issues/195, which is a fun read if you're interested in the process :) This is a work-in-progress overhaul of the core parts of the library to support Spark 3.4+....