Antoine Pitrou
Antoine Pitrou
Also, there are test failures in Python now :-)
> mainly because there is a limit on number of crypto operations with one key What is the theoretical limit, assuming a 256-bit AES key? Also, if column key encryption...
> ~ a billion crypto operations (meaning a billion parquet pages) Out of curiosity, what is the underlying cause?
> The AES GCM math - when an attacker has 2^32 blocks, encrypted with the same key (and random nonces) - the key can be derived from this information. Since...
cc @fjetter
Ok, so it seems the JSON parsing step that takes most of the time in the profile graphs may be spent parsing this JSON string hardcoded (!!) in the SDK's...
> How fast are you wanting/expecting the S3Client to instantiate? You shouldn't be needing to instantiate the that often as it can be reused. I'll let @fjetter elaborate on their...
For the record, here's the current prototype that seems to work on our CI. I ended up caching endpoint providers based on the S3 client configuration's relevant options (the ones...
> In the short term I would recommend to use a single endpoint resolver for all of you s3clients. You can do this by overloading the client when you initialize...
@clee704 Could you please submit a test file to https://github.com/apache/parquet-testing instead of adding it in this PR?