QP Hou

Thanks @Dandandan for filing this, FYI @blogle in case you run into vacuum performance issue with your PoC.

If only there is a rust distributed batch compute framework that we can leverage here ;) On a serious note, I fully agree with you that we should parallelize the...

Good call @Dandandan , filed https://github.com/delta-io/delta-rs/issues/394 for the batch delete support. With batch delete, we should be able to get it down from 1 hour to minutes!

When a lock is repaired by another worker and the current worker resumes, we should still consider the commit successful right? What difference does it make if a specific rename...

@nfx based on https://github.com/delta-io/delta/blob/master/PROTOCOL.md#commit-provenance-information, commit info action is supposed to be schemaless and can store arbitrary json object right?

Based on Serge's comment, we also need to use the same uuid for all files from the same batch, otherwise the counter won't provide any value.

I think that's a good idea for read only streaming use-case :) The log stream abstraction can expose both pull and push based interfaces. Then the existing DeltaTable implementation can...

Just to make sure we are talking about pull/push at the same abstraction level. In my first comment, I was referring to pull and push interfaces at the application level....

@dispanser good call, I filed #128 to track this.

@pitshou could you also the CPU architecture with us? Pasting the output from `uname -a` here should be good enough. We support python version >= 3.6.