Steve Loughran

HADOOP-18184. S3A prefetch unbuffer * Lots of statistic collection with use in tests. * s3a prefetch tests all moved to prefetch. package * and split into caching stream and large...

Yes, this is a lot more than just unbuffer, but its the first time i've really had the code in the IDE with me writing tests to use IOStats, context...

timeout in lru tests ``` [ERROR] Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 843.577 s

tested, s3 london, with `-Dparallel-tests -DtestsThreadCount=8 -Dprefetch -Dscale` and no VPN in the way. This is getting back to as slow as it used to be -and so needs work....

updated patch. the test that caching is failing because files aren't being added to the buffer dir. theory: they are going somewhere else. trivia; rebased patch wouldn't push to the...

testing: some failures. also a timeout in ITestS3APrefetchingLruEviction which I think shows a test in need of some tuning

@virajjasani yes, with that pr. it doesn't happen standalone, but does in a parallel with threads=8 run

...just catching up on this; not ready for merge

There's some race condition with the list add/evict causing intermittent failures of one of the tests. Looks like the failure condition is * block has just been evicted * read()...

s3 london ``` time mvit -Dparallel-tests -DtestsThreadCount=10 -Dscale -Dfs.s3a.root.tests.enabled=false -Djob.id=001 ``` where mvit is mvn verify with some options need to run a few more times to make sure all...