ipex-llm icon indicating copy to clipboard operation
ipex-llm copied to clipboard

DLlib: add enable_hdfs_load decorator

Open lalalapotter opened this issue 1 year ago • 1 comments

Description

According to the customer feedback, there is no SparkContext error while loading HDFS file in ray workers, which is because users use enable_multi_fs_load decorator to load HDFS file and need SparkContext to get gateway of Scala service in each worker. This PR is to add enable_hdfs_load decorator, which is totally implemented in Python, to address the customer issues.

lalalapotter avatar Aug 09 '22 07:08 lalalapotter

Need to add a unit test

hkvision avatar Aug 09 '22 12:08 hkvision