ipex-llm icon indicating copy to clipboard operation
ipex-llm copied to clipboard

Published 20 hours ago • verified publisher icon intel-analytics

If covert a spark df to shards with empty partition, will throw exception

Open dding3 opened this issue 2 years ago • 2 comments

Description

Fix https://github.com/intel-analytics/BigDL/issues/5877. Add logic to check if there is empty partition of spark dataframe, if there is and if spark_df_to_pd_sparkxshards is called, will throw exception.

dding3 avatar Sep 22 '22 17:09 dding3

Why not create an empty spark dataframe partition?

jason-dai avatar Sep 22 '22 23:09 jason-dai

Why not create an empty spark dataframe partition?

You mean create empty pandas df for empty spark dataframe partition? I think it may cause potential problems in the further processing(eg. filter) and the error message may be confusing? Also I think it may bring extra overheads for many empty partitions?

dding3 avatar Sep 23 '22 02:09 dding3