[WIP] POC of runtime module

[odysa]

link #124 Create a runtime module exposing functions like spawn

Example usage:

     let tasks = manifest_list
            .entries()
            .iter()
            .map(|manifest| {
                let cloned_manifest = Arc::new(manifest.clone());
                let file_io = self.file_io.clone();
                async move { cloned_manifest.load_manifest(file_io).await }
            })
            .map(runtime::spawn)
            .collect_vec();

        let manifests: Vec<Manifest> = try_join_all(tasks).await?;

[liurenjie1024]

cc @odysa Is this pr ready for review?

[odysa]

Sorry, my bad. I thought I marked this PR ready for review, but I didn't remove the [WIP]. There are some code changes needed to resolve conflicts. Let me know if you have any comments about the idea of the runtime module. It should be good for review.

There is a struct JoinHandle wrapping handlers from different async runtimes. Implement the JoinHandleExt trait for them so they fit in this wrapper.

The poll from async-std returns Poll<T>, but that from tokio returns Poll<Result<T, JoinError>>. This is why we need to wrap them in our own poll.

[liurenjie1024]

cc @odysa Are you still interested in working on this?

[odysa]

cc @odysa Are you still interested in working on this?

I just got back from vacation today. Let me start working on it.

[liurenjie1024]

cc @odysa Are you still interested in working on this?

I just got back from vacation today. Let me start working on it.

Cool, thanks!

[odysa]

Made code change. Now tokio is always a dependency since traits like AsyncWrite are used. Do you have any plan for them? I made tokio a default feature to avoid breaking change.

[liurenjie1024]

Made code change. Now tokio is always a dependency since traits like AsyncWrite are used. Do you have any plan for them? I made tokio a default feature to avoid breaking change.

I think latest main has removed dependencies on tokio? See https://github.com/apache/iceberg-rust/blob/3b8121eaa9e9628536093836dcc41119716afd9e/crates/iceberg/src/io.rs#L296

But we still have dependency of it here: https://github.com/apache/iceberg-rust/blob/3b8121eaa9e9628536093836dcc41119716afd9e/crates/iceberg/src/writer/file_writer/parquet_writer.rs#L279

[odysa]

Now we run tests by --all-features https://github.com/apache/iceberg-rust/blob/854171d42199f756d2ad1a81a742ce61d9299f05/.github/workflows/ci.yml#L73

Maybe we can have specific tests for tokio and async-std?

[liurenjie1024]

Now we run tests by --all-features

https://github.com/apache/iceberg-rust/blob/854171d42199f756d2ad1a81a742ce61d9299f05/.github/workflows/ci.yml#L73

Maybe we can have specific tests for tokio and async-std?

I think currently a no-default-features test with async-std and a service-fs would be fine

[odysa]

I did cargo test --no-fail-fast --all-targets --no-default-features --features "async-std" --features "storage-fs" --workspace

[Xuanwo]

Thanks @odysa , generally it LGTM. But I think we should resolve #418 first to unblock this.

Got it. I will work on #418 first.

[liurenjie1024]

cc @odysa #418 has been resolved, would you mind to update the pr?