kotlin-spark-api
kotlin-spark-api copied to clipboard
Encoding for Scala Option and Optional
While Dataset<Option<*>> works, whereever a schema needs to be created, such as Dataset<Pair<*, Option<*>>, then it fails.
Option is a Product but is handled as if it doesn't exist in Scala. Meaning Option<T> is encoded as T.
This needs to be looked into