QP Hou

yeah +1 on incrementally migrate to object_store_rs

This is really nice improvement, thanks @wjones127 for picking it up!

Very cool to hear about that as well! Like you mentioned, we haven't even started any optimization work yet :D First obvious step is to parallelizing it with multiple threads....

+1 for me to push the rust data eng community towards a single feature rich and highly optimized object store crate!

> One important difference between our implementation and object_store_rs is that there is that it implements rename_if_not_exists with copy_if_not_exists and delete for all stores. The only ones we implement as...

@ritchie46 let me know what you think about this approach. The core library is now fully decoupled from the arrow-rs crate and only depends on parquet2 for checkpoint parsing. As...

Sounds good @ritchie46 , I will complete the parquet parsing support for map and list this weekend. But the branch I have here right now should be enough to unblock...

@andrei-ionescu I have implemented all the data types other than map and nested list, so it's very close to be complete. However, my time is limited now, so progress will...

+1 on @wjones127 's explicit table registration proposal :) The separation of concerns here also avoids unnecessary overhead in table creation when users don't need to register it in the...

Thank you @roeap for picking up this work! I will take a closer look at it this weekend. :+1: from me for including datafusion-expr if needed.