Steve Loughran comments

Results 315 comments of


                                            Steve Loughran

Unable to connect to local Minio server when a proxy configuration is specified, even when the Minio server host is set in the http.nonProxyHosts.

> all the http clients supported by the SDK will honor proxy system properties - that's the expectation at least. If the latest SDK version is not picking them up...

Pure Java NIO LocalInputFile Implementation

Just going to highlight that hadoop 3.4.0 + supports scatter/gather IO for local, s3 connections. If you do go with this you should benchmark the work and consider pulling up...

DRAFT: Alternative Strawman proposal for a new V3 footer format in Parquet

> (e.g. if they happen to be exposing thrift structures as a public API) what do we know here about this? which apps do/don't?

Add more constructors to ParquetFileReader

Can I suggest the builder pattern, allows for more evolution over time and allows every option to have a default value

MAPREDUCE-7465. Add support for parallelism in FileOutputCommiter via 'mapreduce.fileoutputcommitter.parallel.threshold'

@Arnaud-Nauwynck just stuck up #6399 which is rajesh's impl with my reviews in too. I'm not going to merge that into hadoop either because it's still got so many problems...

should not use seek() for skipping very small column chunks. better to read and ignore data.

+ @mukund-thakur * I think we should revisit that min seek range, at least add a good default for ABFS, and make it configurable. File a HADOOP- JIRA, ideally with...

should not use seek() for skipping very small column chunks. better to read and ignore data.

I think we could do some micro benchmarking here as well. What we wantis have a skip size such that it is faster to discard the data it is to...

[Core][Spark] Improve DeleteOrphanFiles action to return additional details of deleted orphan files

> One of the common causes of delete failure in the public cloud is hitting the API quotas and unnecessary re-runs of the delete action Not really. What size of...

Multipart upload failure from crtBuilder() with S3 like Storage ex S3 Cohesity

This looks dangerously like #5415 where again, we see 400 on a retry of a partial file upload. This is only triggered after a 500; maybe this third-party store has...

Use SupportsPrefixOperations for Remove OrphanFile Procedure

HadoopFileIO (and therefore the local fs) supports listPrefix. It'll need a CustomFileIO as with similar tests