ipex-llm
ipex-llm copied to clipboard
DLlib: add enable_hdfs_load decorator
Description
According to the customer feedback, there is no SparkContext
error while loading HDFS file in ray workers, which is because users use enable_multi_fs_load
decorator to load HDFS file and need SparkContext
to get gateway of Scala service in each worker. This PR is to add enable_hdfs_load
decorator, which is totally implemented in Python, to address the customer issues.
Need to add a unit test