iceberg-rust icon indicating copy to clipboard operation
iceberg-rust copied to clipboard
verified publisher icon apache

feat(storage): Add hdfs support via webhdfs

Open Xuanwo opened this issue 9 months ago • 2 comments

Which issue does this PR close?

Part of https://github.com/apache/iceberg-rust/issues/1130

What changes are included in this PR?

Add webhdfs based hdfs storage support in iceberg-rust.

Are these changes tested?

Webhdfs service itself is tested at opendal side, maybe we can setup related tests in iceberg-rust too.

Xuanwo avatar Mar 24 '25 04:03 Xuanwo

Thanks @Xuanwo with this pr. There are two problems with feature based design:

  1. User could only choose hdfs backend at compile time, which seems inflexible to me. For example, I'm integrating iceberg-rust with a compute engine like trino, which may access different storages at runtime.
  2. It may not be transparent to user. IIRC, webhdfs's configuration and url schema are different from native hdfs library.

liurenjie1024 avatar Mar 25 '25 01:03 liurenjie1024

Maybe we can write down our choice and discuss at community sync meeting?

liurenjie1024 avatar Mar 25 '25 01:03 liurenjie1024

Closing this one since we won't introduce webhdfs support in iceberg-rust for now.

Xuanwo avatar May 14 '25 10:05 Xuanwo