Batch listFiles API's for GoogleHadoopFileSystem

[naveenkrdremio]

Fixes https://github.com/GoogleCloudDataproc/hadoop-connectors/issues/565

Changes:

1. Implement listFiles() API to return the files in iterative way.
2. Enhance listFileInfoForPrefixPage() to support non-recursive listing.

[google-cla[bot]]

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

:memo: Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

---

What to do if you already signed the CLA

Individual signers

• It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

• Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
• The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
• The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

[naveenkrdremio]

@googlebot I signed it!

[google-cla[bot]]

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

:memo: Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

---

What to do if you already signed the CLA

Individual signers

• It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

• Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
• The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
• The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

[jaceklaskowski]

Is this a duplicate of https://github.com/GoogleCloudDataproc/hadoop-connectors/pull/584?

[naveenkrdremio]

Is this a duplicate of #584?

Yes . Will close the other one #584.

[naveenkrdremio]

@googlebot I signed it!

[google-cla[bot]]

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

:memo: Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

---

What to do if you already signed the CLA

Individual signers

• It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

• Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
• The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
• The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

[naveenkrdremio]

@googlebot I signed it!

[naveenkrdremio]

@jaceklaskowski Can you review please ?

[mprashanthsagar]

/gcbrun

[codecov[bot]]

Codecov Report

Merging #585 (e39b052) into master (cef25d6) will decrease coverage by 11.66%. The diff coverage is 91.89%.

:exclamation: Current head e39b052 differs from pull request most recent head 946b284. Consider uploading reports for the commit 946b284 to get more accurate results

@@              Coverage Diff              @@
##             master     #585       +/-   ##
=============================================
- Coverage     80.29%   68.63%   -11.67%     
+ Complexity     2035     1745      -290     
=============================================
  Files           138      138               
  Lines          8913     8942       +29     
  Branches       1056     1060        +4     
=============================================
- Hits           7157     6137     -1020     
- Misses         1319     2384     +1065     
+ Partials        437      421       -16     

Flag	Coverage Δ
hadoop2integrationtest	?
hadoop2unittest	68.54% <91.89%> (+0.13%)	:arrow_up:
hadoop3integrationtest	?
hadoop3unittest	68.58% <91.89%> (+0.13%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...loud/hadoop/fs/gcs/GoogleHadoopFileSystemBase.java	75.29% <86.95%> (-12.84%)	:arrow_down:
...oud/hadoop/gcsio/GoogleCloudStorageFileSystem.java	86.35% <100.00%> (-4.38%)	:arrow_down:
...gle/cloud/hadoop/gcsio/cooplock/CoopLockUtils.java	0.00% <0.00%> (-100.00%)	:arrow_down:
.../cloud/hadoop/gcsio/testing/TestConfiguration.java	0.00% <0.00%> (-100.00%)	:arrow_down:
...d/hadoop/gcsio/cooplock/CoopLockOperationType.java	0.00% <0.00%> (-100.00%)	:arrow_down:
...ud/hadoop/io/bigquery/AvroBigQueryInputFormat.java	0.00% <0.00%> (-100.00%)	:arrow_down:
...a/com/google/cloud/hadoop/fs/gcs/CoopLockFsck.java	0.00% <0.00%> (-88.47%)	:arrow_down:
...adoop/gcsio/cooplock/RenameOperationLogRecord.java	0.00% <0.00%> (-87.50%)	:arrow_down:
...loud/hadoop/gcsio/cooplock/CoopLockRecordsDao.java	0.00% <0.00%> (-87.20%)	:arrow_down:
...google/cloud/hadoop/fs/gcs/CoopLockFsckRunner.java	0.00% <0.00%> (-85.78%)	:arrow_down:
... and 35 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update eaaebef...946b284. Read the comment docs.

[naveenkrdremio]

@mprashanthsagar , I have incorporated the comments. Please review. Have not added Implementations for https://github.com/apache/hadoop/blob/b6c06c4b76ca5c206639e2ddea4f3bef393dbb11/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java#L2175. Is in progress.

[naveenkrdremio]

@medb , Please review latest patch.