Peter Ke

~~We're also seeing a similar issue where constructing the `DeltaTable` takes 20-30 seconds, which is longer than we expected.~~ ~~For context, we're using version 0.17.1. Our table is on AWS...

@ion-elgreco @rtyler thanks for the suggestions! I will look into the `QueryBuilder` approach. Just so I understand correctly, the general idea is that we leave functionality like projecting/filtering/joining to datafusion...

Hey @rtyler @ion-elgreco I took a shot implementing the `QueryBuilder` approach. IMO this would still be pretty valuable for people who don't want to manage a separate datafusion-python dependency. Would...

Thanks, I added the experimental warning to `QueryBuilder`

@roeap looks cool! Glad you're already working on it 😄 I'm curious about the decision to only cache json log files. What's fundamentally blocking your implementation from also caching checkpoints...

> @PeterKeDer - there may be a few things to unpack before that choice makes sense 😆. > > 1. we have 2 phases to look at a) metadata b)...

@roeap @ion-elgreco I cleaned this up & added simple tests, appreciate if you can take a look and let me know your thoughts

@roeap I took a shot at moving the config over to `DeltaConfig`, let me know if I'm using it correctly. Also, I noticed `decorate_store` is not being applied to the...

@roeap were you able to take a look at the test failures? I wonder if there are differences writing files when the test is running in the pipeline vs locally...

Ahh interesting, thanks for looking into it! I did use `tempdir` in some other tests though, I wonder why they didn't fail 🤔 The change to make the store more...