Steve Loughran

this is consistent with hadoop FS spec, which says when `create(path)` returns, `exists(path)` must hold https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/filesystem/filesystem.html#FSDataOutputStream_create.28Path.2C_....29 the fact that s3 doesn't do this is a divergence there. for atomicity, write...

we have an explicit API for multipart uploads, with conformance tests: https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/multipartuploader.md this is sufficient for us to reimplement the S3A commit algorithm through a public API...which would give GCS...

looking forward to it. FWIW, that multipart API could also be used for a high performance parallelised distcp tool, where blocks were uploaded in parallel from multiple source workers and...

sort out some azure creds. good to play with the opposition anyway

also, be good to know what linux version you are testing on. i will have to test this on a vm before merging too, because the openssl native integration has...

ok. i will do an (ec2) linux run later. that or fix my pi400 up for testing, which might be fun anyway

Ask on the HDEFS mailing list; they are the ones that get unhappy first when things break.

read my profile.

+ @mukund-thakur can you look at this?

ready for review again @mukund-thakur