Wenchen Fan comments

Results 245 comments of


                                            Wenchen Fan

[SPARK-46654][SQL] Make to_csv can correctly display complex types data

why do we remove a working feature? what's wrong with `to_csv` generating non-standard but pretty strings for these values?

[SPARK-46654][SQL] Make to_csv can correctly display complex types data

So we make a breaking change simply because we don't like the object address in the string? can we fix it?

[SPARK-46654][SQL] Make to_csv can correctly display complex types data

You never know how people will use Spark and I'm not sure why it's necessary to ban this use case. Having inconsistency between scala and python is bad. Can we...

[SPARK-46654][SQL] Make to_csv can correctly display complex types data

I'd prefer to fix the inconsistency. Can we use `ToPrettyString.eval` to generate pretty strings for these types in `to_csv`?

[SPARK-46654][SQL] Make to_csv can correctly display complex types data

It's ok to not support round-trip. array/struct/map is not CSV standard anyway. I think the principle is to avoid breaking changes if possible. Since `to_csv` already supports array/struct/map, let's keep...

[SPARK-46654][SQL] Make to_csv can correctly display complex types data

If no objections, I'll revert this tomorrow. We can have a new PR to add pretty string for array/struct/map in `to_csv`.

[SPARK-46617][SQL] Create-table-if-not-exists should not silently overwrite existing data-files

I agree that the current behavior is buggy, but we should have more discussion about the expected behavior. For `CREATE TABLE IF NOT EXISTS`, I think the user expectation is...

[SPARK-48162][SQL] Add collation support for MISC expressions

thanks, merging to master!

[SPARK-48155][SQL] AQEPropagateEmptyRelation for join should check if remain child is just BroadcastQueryStageExec

thanks, merging to master!

[SPARK-48241][SQL] CSV parsing failure with char/varchar type columns

thanks, merging to master/~3.5~!