Lee Yang

add support for custom data loading (e.g. NVTabular) in KerasEstimator

3

## Checklist before submitting - [x] Did you read the [contributor guide](https://github.com/horovod/horovod/blob/master/CONTRIBUTING.md)? - [x] Did you update the docs? - [x] Did you write any tests to validate this change?...

Currently, the Spark integration for PyTorchLightning [TorchEstimator supports customizable data_modules](https://github.com/horovod/horovod/blob/9d56c5a6d55e9619e100d19480a11952d116f4b3/horovod/spark/lightning/estimator.py#L109-L110), but the [RemoteTrainer for the KerasEstimator is hard-coded to use petastorm](https://github.com/horovod/horovod/blob/9d56c5a6d55e9619e100d19480a11952d116f4b3/horovod/spark/keras/remote.py#L112). Ideally, the KerasEstimator should support customizable data modules/loaders as...

[SPARK-40264][ML] add batch_infer_udf function to pyspark.ml.functions

2

### What changes were proposed in this pull request? Add a `batch_infer_udf` function to `pyspark.ml.functions` to help users construct a `pandas_udf` for ML/DL model inference in Spark. This UDF adds...

Follow-up to #3602, to add support for customizable data modules/loaders in the TorchEstimator.