SparkXShards to_spark_df only process the first element in one partition

[cyita]

If a partition of an XShards has more than one pandas dataframe, the to_spark_df function will only process the first pdf. Error code: https://github.com/intel-analytics/BigDL/blob/main/python/orca/src/bigdl/orca/data/shard.py#L583

[dding3]

For SparkXShards of pandas dataframe, it's by design there is only one pandas dataframe for each partition.

[jason-dai]

Is this accurate? Is it possible to have union type operations?

If this is accurate, we still need to check and report error.

[dding3]

I think currently user can create sparkxshards of pandas dataframe in 2 ways:

1. use orca api read_csv which will internally create rdd of pandas df and it will create 1 pandas df per partition
2. user call rdd mappartion operations to create sparkxshards of pandas dataframe, it seems not common?

[jason-dai]

I think currently user can create sparkxshards of pandas dataframe in 2 ways:

1. use orca api read_csv which will internally create rdd of pandas df and it will create 1 pandas df per partition
2. user call rdd mappartion operations to create sparkxshards of pandas dataframe, it seems not common?

The user can always call XShards operations (e.g., transform_shard) to create a new XShards

[dding3]

Yes, user can do that. I think in that case, if users want to create multiple pandas dataframes with one partition, these dataframes may have different schema, otherwise why mutiple dataframes with same schema other than one.

[jason-dai]

Yes, user can do that. I think in that case, if users want to create multiple pandas dataframes with one partition, these dataframes may have different schema, otherwise why mutiple dataframes with same schema other than one.

We should either support that, or report error; silent failure is bad user experience.

[dding3]

Sure, will add check and report error.