iceberg-rust icon indicating copy to clipboard operation
iceberg-rust copied to clipboard

test: Add S3 partition directory verification in integration test

Open hackintoshrao opened this issue 11 months ago • 2 comments

Enhances append_partition_data_file_test.rs to verify the correct partition layout in MinIO:

  • Add URL parsing to extract bucket and prefix from table location
  • Implement S3 ListObjectsV2 check for partition directory
  • Add debug logging for table location and data file paths
  • Verify objects exist in the expected partition directory (id=100/)

This helps diagnose the partition layout issue by confirming whether files are being written to the correct partitioned directory structure.

hackintoshrao avatar Jan 13 '25 01:01 hackintoshrao

Thanks @hackintoshrao for this pr. I'll convert it to draft first as it's only used for showing the expected path, but there is no guarantee in iceberg spec that it should generate such a directory.

liurenjie1024 avatar Jan 24 '25 09:01 liurenjie1024

Thanks @hackintoshrao for this pr. I'll convert it to draft first as it's only used for showing the expected path, but there is no guarantee in iceberg spec that it should generate such a directory.

Yes, I feel the same as @liurenjie1024. The physical layout is not part of the iceberg spec, and the writer has the ultimate authority to decide how to place them. It's better not to enforce this here.

Also, cc @Fokko—what do you think?

Xuanwo avatar Mar 06 '25 17:03 Xuanwo