test: Add S3 partition directory verification in integration test
Enhances append_partition_data_file_test.rs to verify the correct partition layout in MinIO:
- Add URL parsing to extract bucket and prefix from table location
- Implement S3 ListObjectsV2 check for partition directory
- Add debug logging for table location and data file paths
- Verify objects exist in the expected partition directory (id=100/)
This helps diagnose the partition layout issue by confirming whether files are being written to the correct partitioned directory structure.
Thanks @hackintoshrao for this pr. I'll convert it to draft first as it's only used for showing the expected path, but there is no guarantee in iceberg spec that it should generate such a directory.
Thanks @hackintoshrao for this pr. I'll convert it to draft first as it's only used for showing the expected path, but there is no guarantee in iceberg spec that it should generate such a directory.
Yes, I feel the same as @liurenjie1024. The physical layout is not part of the iceberg spec, and the writer has the ultimate authority to decide how to place them. It's better not to enforce this here.
Also, cc @Fokko—what do you think?