Steve Loughran
Steve Loughran
Boost performance reading from object stores in hadoop 3.3.0+ by using the openFile builder API and passing in the file length as an option (can save a HEAD) and asks...
if the destination path parent is the same as the source path parent, or is a child of it, there's no need to repair the implicit directory on the source...
while writing some tests which collect FS stats in teardown, my logs are full of stack traces. ``` Jan 21, 2022 2:56:17 PM com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem setHttpStatistics WARNING: Error while getting GCS...
jobId.toString() to only be called when the ID isn't null. this doesn't surface in MR, but spark seems to manage it ### How was this patch tested? through my downstream...
### Description of PR Declares its compatibility with the stream capability "mapreduce.job.committer.dynamic.partitioning" spark will need to cast to StreamCapabilities and then probe. ### How was this patch tested? I have...
This moves the "common" classes for the prefetch code from hadoop-aws to hadoop common under org.apache.hadoop.fs.impl.prefetch ### How was this patch tested? s3 london; all good apart from landsat scale...
New trash policies, and a schema specific trash policy set by fs.SCHEMA.trash.policy. This lets clusters declare different policies for different stores in the same cluster. * `CloudTrashPolicy`: for abfs with...
Changes * Javadocs in ByteBufferPositionedReadable are a bit stricter. * switch to parameterized JUnit test runs for on/off heap buffers * factor put main assertions into higher level asserts New...
This is me looking at what minimal changes could be made to boost IO performance working with the cloud stores. Compiles against hadoop 3.3.3; will need hadoop 3.3.5 for some...
Remove the deprecated classes PathGlobPattern and DeprecatedFieldProjectionFilter so that Parquet will compile against hadoop 3.x. If a thrift reader is configured to use the now-deleted filter, by setting the filter...