what's the recommended way to test rest catalog changes?

[Xuanwo]

Hi team, just curious, what's the recommended way to test rest catalog changes? For example, iceberg provides rest docker for local setup: https://github.com/databricks/iceberg-rest-image, which I actually introduce in my docker compose for integration test.

What I'm expecting is something similar to kubernetes envtest, which we could bring up in-process. Curious if we have something similar for iceberg? Thank you!

Originally posted by @dentiny in https://github.com/apache/iceberg-rust/issues/1266#issuecomment-2833715169

[liurenjie1024]

cc @dentiny What kind of changes are you talking about? We have integration tests for rest catalog, see https://github.com/apache/iceberg-rust/blob/30181383a7d8684b08b0fba34821e82c5669dd45/crates/catalog/rest/tests/rest_catalog_test.rs#L77

[liurenjie1024]

This seems like a question, how about we convert it into discussion?

[dentiny]

Hi @liurenjie1024 / @Xuanwo , thank you so much for your help and response! And also apology for the confusion.

My question is:

• Since https://github.com/apache/iceberg-rust/pull/1266 is a bug fix, is it possible to add a unit test / integration test, which fails before the change, but passes after the change?
• The PR mentions to fix namespace_exists, but I don't see tests against this interface (we do have get_namespace)
• Thanks for the test code pointer! It looks like we're rely on docker for integration test, I'm more curious if you guys have recommendation / you know to unit test rest catalog?
  • The reason I mention kubernetes envtest is, I imagine there could be a similar test infrastructure for iceberg rest catalog, which I'm not aware of.

[liurenjie1024]

Since https://github.com/apache/iceberg-rust/pull/1266 is a bug fix, is it possible to add a unit test / integration test, which fails before the change, but passes after the change? The https://github.com/apache/iceberg-rust/pull/1266#issuecomment-2833715169 mentions to fix namespace_exists, but I don't see tests against this interface (we do have get_namespace)

Yes, it would be better if we have added some integration tests when fixing it.

The reason I mention kubernetes envtest is, I imagine there could be a similar test infrastructure for iceberg rest catalog, which I'm not aware of.

No, iceberg rest catalog is just a spec, and we only use docker to setup a ref implementation. You can find uts here: https://github.com/apache/iceberg-rust/blob/f162400af948628ab6f25ddf6b148ed87662328e/crates/catalog/rest/src/catalog.rs#L886

[Fokko]

I would not use: https://github.com/databricks/iceberg-rest-image, instead use https://hub.docker.com/r/apache/iceberg-rest-fixture/tags. This is now part of the Iceberg repository: https://github.com/apache/iceberg/tree/main/docker/iceberg-rest-fixture

It builds a nightly container from the latest release, and it also pins on each of the Java releases. This makes it easy to test out the latest features, such as deletion vectors and row-lineage.

[liurenjie1024]

I would not use: https://github.com/databricks/iceberg-rest-image, instead use https://hub.docker.com/r/apache/iceberg-rest-fixture/tags. This is now part of the Iceberg repository: https://github.com/apache/iceberg/tree/main/docker/iceberg-rest-fixture

It builds a nightly container from the latest release, and it also pins on each of the Java releases. This makes it easy to test out the latest features, such as deletion vectors and row-lineage.

I think we have changed it: https://github.com/apache/iceberg-rust/blob/83ca98fd55de9485eaa63cc00295876d25ee0149/crates/integration_tests/testdata/docker-compose.yaml#L23 https://github.com/apache/iceberg-rust/blob/83ca98fd55de9485eaa63cc00295876d25ee0149/crates/catalog/rest/testdata/rest_catalog/docker-compose.yaml#L23

[liurenjie1024]

I think the question has been answered, and I'll close it for now. Feel free to reopen it if you still have questions.