Nico de Vos

[WIP] Scala 2.12 / Spark 3 upgrade

14

**Related issues** https://github.com/salesforce/TransmogrifAI/issues/336 https://github.com/salesforce/TransmogrifAI/issues/332 **Describe the proposed solution** Upgrade to Scala 2.12 and Spark 3 **Describe alternatives you've considered** Living in the past, suffering from security issues and missing out...

The Boston housing data is problematic, see discussion here on the scikit-learn issue tracker: https://github.com/scikit-learn/scikit-learn/issues/16155 Following scikit-learn, we could replace it with a different data set (not necessarily housing).

"WindowExec: No Partition Defined for Window operation!" warnings on Spark EntitySets

1

Both when I add Spark DataFrames to my EntitySet and when I call `.dfs()` on the Spark EntitySet, I see a flood of warnings: ```22/04/26 16:41:09 WARN WindowExec: No Partition...

Add util with cluster performance metrics

1

See Bai et al., 2011 for details: - Category utility function - Adjusted rand index (WON'T DO: `sklearn.metrics` already implements this) - Set matching error (WON'T DO: seems obscure)

Java / Scala bindings for Polars?

6

### Problem description Are there ideas or plans around bindings for JVM-based languages like Java, Kotlin or Scala? Especially the latter might be of interest in combination with Spark. Users...

Problem decoding union of map: Failed to convert JSON to Avro

2

Example schema: ``` protocol ScoreData { record ScoreData { string id; union {null, map} multiScore = null; } } ``` Example payload: ```"{\"id\":\"myKey\",\"multiScore\":{\"map\":{\"a\":10.0}}}"``` Exception: ```tech.allegro.schema.json2avro.converter.AvroConversionException: Failed to convert JSON to...