Pavel Solodovnikov

Force-pushed the branch to fix some build issues on macOS and Windows. Changelog can be found here: https://github.com/apache/arrow/compare/b6714018c4384cd00f0b9a4e92d804671d1d381b..3f2b141dea8ded351d41d687245917821295277a?diff=unified

I believe appveyor failures don't have anything to do with my patches... Also, "Dev / Source Merge Script" jobs seem to fail due to some issues related to 10.0.0 freeze?

Force-pushed the branch to address review comments. Diff can be found here: https://github.com/apache/arrow/compare/3f2b141dea8ded351d41d687245917821295277a..136ae80540f851219c8d9a920470d4863a428d30 Changelog: * Rename `partitions_readahead` to `directory_readahead`, as well as the methods in `AsyncStatSelector` to get rid of...

> Thanks for the update, I posted a couple comments. > > Also, can you add some localfs-specific tests for this? Ideally you would stress both with and without directory...

Force-pushed the branch to address review comments from @pitrou and @westonpace . The diff can be found here: https://github.com/apache/arrow/compare/136ae80540f851219c8d9a920470d4863a428d30..ac2954d39b5de052b3c05563d290f00c23b51b34, also pushed a fixup to amend commit message. Changelog: * Change...

Force-pushed the branch to address review comments from @pitrou The diff can be found there: https://github.com/apache/arrow/compare/affcca7fa308c897ff5a358d1668486abed6c5fa..09142855f750ea046fb081c17d23b5fa675f7e9f Changelog: * Removed `AutoClosingProducer` in favor of a more simplistic and correct `std::shared_ptr` approach...

@pitrou Your numbers are in line with what I get on my machine. I agree we can tune the defaults to be `kDefaultDirectoryReadahead = 16` and `kDefaultFileInfoBatchSize = 1000`.

Force-pushed the branch to address some follow-up comments. The diff can be found here: https://github.com/apache/arrow/compare/09142855f750ea046fb081c17d23b5fa675f7e9f..eb77fad86834a6c10244eb924197e949b7faba3f Changelog: * Set `kDefaultDirectoryReadahead` to 16 instead of 1, based on benchmarking results * Fixed...

Force-pushed the branch, addressed style comments from @westonpace. Diff can be found here: https://github.com/apache/arrow/compare/fae325e7c553cc857ae9d05c757d5ff90e646260..da4235e3c08b46a8ff7a5207a93c03db75ce5515

Rebased and force-pushed the branch. Unfortunately, I cannot attach a link to the diff because rebasing broke it. Changes summary: * Added `utils/async_generator_fwd.h` * `GetFragmentsAsyncImpl()` forwards `GetFragmentsImpl()` to `MakeBackgroundGenerator() +...