Chenyang Liu

Is this project still actively maintained? There's some Spark upgrade which is good. but from the contribution insights and commit history looks like we don't have an active maintainer anymore.

SimpleIon load performance: 7min on a 30Mb ion text files

15

Hi, it looks like there's a big performance issue with the simpleion load/load function. I did a little experiment: - 30 MB ion text: 7min to load into memory -...

CLI support for encoding/importing from JSON

2

### Things to check first - [X] I have searched the existing issues and didn't find my feature already requested there ### Feature description Currently the CLI tool only supports...

See https://spark.apache.org/docs/3.4.0/api/python/reference/api/pyspark.ml.functions.predict_batch_udf.html and https://developer.nvidia.com/blog/distributed-deep-learning-made-easy-with-spark-3-4/. The current implementation seems to be subject to issues below. > The predict_batch_udf introduces standardized code for: > > - Translating Spark DataFrames into NumPy arrays,...