jmpanfil

Speeding up loading data from spark

3

I've been working on using petastorm to train PyTorch models from spark dataframes (somewhat following [this guide](https://docs.databricks.com/applications/machine-learning/load-data/petastorm.html)). I'm curious if there are any ways I can speed up data loading....

Drawbacks of using this library

1

I love pipes in R, so this is a very enticing option for me. However, what are the main drawbacks of using this library? I imagine: - It's not "pythonic"...

Using Python trained LightGBM model to predict on Spark DataFrame (in Databricks)

1

I am curious if there is a straightforward way to use a regular python trained LightGBM model on a distributed Spark DataFrame. The model was trained using version 4.3.0 and...