HADOOP-19139.No GetPathStatus for opening AbfsInputStream

[saxenapranav]

jira: https://issues.apache.org/jira/browse/HADOOP-19139

Read API gives contentLen and etag of the path. This information would be used in future calls on that inputStream. Prior information of eTag is of not much importance.

Added two configs: 1.fs.azure.input.stream.lazy.open.optimization.enabled : if this is true, head call would be prevented for opening AbfsInputStream. - default: false, as in trunk right now, head call always happen before opening AbfsInputStream. 2. fs.azure.prefetch.on.first.read.enabled: if this is false, the first read call on AbfsInputStream would not execute prefetch. But, after first read, subsequent calls would be able to call prefetches.

• default: true, as in trunk right now, prefetches are tried for each read on AbfsInputStream/

TESTS done on :

1. default configs 2.fs.azure.input.stream.lazy.open.optimization.enabled = true; fs.azure.prefetch.on.first.read.enabled = false :::: AGGREGATED TEST RESULT ::::

HNS-OAuth

[INFO] Results: [INFO] [WARNING] Tests run: 137, Failures: 0, Errors: 0, Skipped: 2 [INFO] Results: [INFO] [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:329 » TestTimedOut test timed o... [INFO] [ERROR] Tests run: 578, Failures: 0, Errors: 1, Skipped: 82 [INFO] Results: [INFO] [ERROR] Errors: [ERROR] ITestAbfsTerasort.test_120_terasort:262->executeStage:206 » IO The ownership o... [INFO] [ERROR] Tests run: 380, Failures: 0, Errors: 1, Skipped: 55

HNS-SharedKey

[INFO] Results: [INFO] [WARNING] Tests run: 137, Failures: 0, Errors: 0, Skipped: 3 [INFO] Results: [INFO] [ERROR] Failures: [ERROR] ITestAzureBlobFileSystemRandomRead.testValidateSeekBounds:278->Assert.assertTrue:42->Assert.fail:89 There should not be any network I/O (elapsedTimeMs=72). [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:329 » TestTimedOut test timed o... [INFO] [ERROR] Tests run: 585, Failures: 1, Errors: 1, Skipped: 37 [INFO] Results: [INFO] [WARNING] Tests run: 380, Failures: 0, Errors: 0, Skipped: 41

NonHNS-SharedKey

[INFO] Results: [INFO] [WARNING] Tests run: 137, Failures: 0, Errors: 0, Skipped: 9 [INFO] Results: [INFO] [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:344->lambda$testAcquireRetry$6:345 » TestTimedOut [ERROR] ITestAzureBlobFileSystemLease.testTwoWritersCreateAppendWithInfiniteLeaseEnabled:186->twoWriters:154 » TestTimedOut [INFO] [ERROR] Tests run: 569, Failures: 0, Errors: 2, Skipped: 267 [INFO] Results: [INFO] [WARNING] Tests run: 380, Failures: 0, Errors: 0, Skipped: 44

AppendBlob-HNS-OAuth

[INFO] Results: [INFO] [WARNING] Tests run: 137, Failures: 0, Errors: 0, Skipped: 2 [INFO] Results: [INFO] [ERROR] Failures: [ERROR] ITestAzureBlobFileSystemRandomRead.testSkipBounds:222->Assert.assertTrue:42->Assert.fail:89 There should not be any network I/O (elapsedTimeMs=703). [ERROR] Errors: [ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:329 » TestTimedOut test timed o... [INFO] [ERROR] Tests run: 596, Failures: 1, Errors: 1, Skipped: 82 [INFO] Results: [INFO] [ERROR] Errors: [ERROR] ITestAbfsTerasort.test_120_terasort:262->executeStage:206 » IO The ownership o... [INFO] [ERROR] Tests run: 380, Failures: 0, Errors: 1, Skipped: 55

Time taken: 44 mins 39 secs.