spark-excel
spark-excel copied to clipboard
Error Handling for Corrupt Files in Chunk Processing
Is there an existing issue for this?
- [X] I have searched the existing issues
Current Behavior
I am using the spark-excel library to read Excel files in chunks from a directory. My process involves reading multiple Excel files and then performing operations on the data. However, I've encountered an issue where if any of the files are corrupt or cause an error during reading, the entire processing job fails and does not skip the problematic file.
Expected Behavior
I would expect the library to provide a mechanism to skip over files that cause errors during the read operation, allowing the process to continue with the next files in the list.
Steps To Reproduce
Set up a Spark session and configure it to read Excel files using spark-excel. Use a loop to read files in chunks from a specified directory. If an error occurs while reading a file (e.g., due to corruption or format issues), the process stops, and subsequent files are not processed.
Environment
- Spark version: 3.4.1
- Spark-Excel version: 0.20.3
- OS: Linux
- Cluster environment: Databricks
Anything else?
No response