Szehon Ho

A bit late to the discussion, and didnt look very closely, but I wonder if we can make Testwriters for the various metadata that writes null for all optional fields,...

Was chatting with @dramaticlly about it, cc if you want to look at it.

@RussellSpitzer addressed the comments, thanks!

Actually hold on a second, looking at a small refactor to make it more generic to add a readable_metric definition in future

@RussellSpitzer should be good now for another look when you get a chance, thanks!

Yep , test should be here: https://github.com/apache/iceberg/blob/6681dba9bc7dc0d793aa8de739d2b9962260b0ff/spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/source/TestMetadataTableReadableMetrics.java

Would love to see what is a good way to simplify it without breaking the checks. Currently compares every single field.

Thanks @RussellSpitzer @aokolnychyi @chenjunjiedada for detailed reviews

Closing as it's now broken into smaller prs

Some implementation notes. SPARK-41471 works by making the ShuffleExchangeExec side of the join have a KeyGroupedPartitioning, which is created by the other side's KeyGroupedShuffleSpec and is a clone of it...