hadoop icon indicating copy to clipboard operation
hadoop copied to clipboard

HADOOP-18656: [ABFS] Adding Support for Paginated Delete for Large Directories in HNS Account

Open anujmodi2021 opened this issue 1 year ago • 9 comments

Description of PR

Jira Ticket: https://issues.apache.org/jira/browse/HADOOP-18656

Today, when a recursive delete is issued for a large directory in ADLS Gen2 (HNS) account, the directory deletion happens in O(1) but in backend ACL Checks are done recursively for each object inside that directory which in case of large directory could lead to request time out. Pagination is introduced in the Azure Storage Backend for these ACL checks.

More information on how pagination works can be found on public documentation of Azure Delete Path API.

This PR contains changes to support this from client side. To trigger pagination, client needs to add a new query parameter "paginated" and set it to true along with recursive set to true. In return if the directory is large, server might return a continuation token back to the caller. If caller gets back a continuation token, it has to call the delete API again with continuation token along with recursive and pagination set to true. This is similar to directory delete of FNS account.

Pagination is available only in versions "2023-08-03" onwards. PR also contains functional tests to verify driver works well with different combinations of recursive and pagination features for both HNS and FNS account. Full E2E testing of pagination requires large dataset to be created and hence not added as part of driver test suite. But extensive E2E testing has been performed.

How was this patch tested?

New tests added and existing tests ran.


:::: AGGREGATED TEST RESULT ::::

HNS-OAuth

[INFO] Results: [INFO] [WARNING] Tests run: 141, Failures: 0, Errors: 0, Skipped: 5 [INFO] Results: [INFO] [WARNING] Tests run: 340, Failures: 0, Errors: 0, Skipped: 41

HNS-SharedKey

[INFO] Results: [INFO] [WARNING] Tests run: 141, Failures: 0, Errors: 0, Skipped: 5 [INFO] Results: [INFO] [WARNING] Tests run: 340, Failures: 0, Errors: 0, Skipped: 41

NonHNS-SharedKey

[INFO] Results: [INFO] [WARNING] Tests run: 141, Failures: 0, Errors: 0, Skipped: 11 [INFO] Results: [INFO] [WARNING] Tests run: 589, Failures: 0, Errors: 0, Skipped: 266 [INFO] Results: [INFO] [WARNING] Tests run: 340, Failures: 0, Errors: 0, Skipped: 44

AppendBlob-HNS-OAuth

[INFO] Results: [INFO] [WARNING] Tests run: 141, Failures: 0, Errors: 0, Skipped: 5 [INFO] Results: [INFO] [WARNING] Tests run: 340, Failures: 0, Errors: 0, Skipped: 41

Time taken: 25 mins 38 secs.

anujmodi2021 avatar Jan 04 '24 14:01 anujmodi2021

:broken_heart: -1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 51s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 1s codespell was not available.
+0 :ok: detsecrets 0m 1s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ trunk Compile Tests _
-1 :x: mvninstall 42m 27s /branch-mvninstall-root.txt root in trunk failed.
+1 :green_heart: compile 0m 36s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: compile 0m 34s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: checkstyle 0m 30s trunk passed
+1 :green_heart: mvnsite 0m 38s trunk passed
+1 :green_heart: javadoc 0m 37s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javadoc 0m 32s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: spotbugs 1m 3s trunk passed
+1 :green_heart: shadedclient 32m 17s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 0m 28s the patch passed
+1 :green_heart: compile 0m 28s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javac 0m 28s the patch passed
+1 :green_heart: compile 0m 27s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: javac 0m 27s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
-0 :warning: checkstyle 0m 18s /results-checkstyle-hadoop-tools_hadoop-azure.txt hadoop-tools/hadoop-azure: The patch generated 2 new + 6 unchanged - 0 fixed = 8 total (was 6)
+1 :green_heart: mvnsite 0m 30s the patch passed
+1 :green_heart: javadoc 0m 25s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javadoc 0m 25s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: spotbugs 1m 2s the patch passed
+1 :green_heart: shadedclient 32m 45s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: unit 2m 1s hadoop-azure in the patch passed.
+1 :green_heart: asflicense 0m 35s The patch does not generate ASF License warnings.
123m 3s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/1/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6409
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux d12aa8aef27c 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 4580d3b12e620c2185f9096dac18f1fe60c90949
Default Java Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/1/testReport/
Max. process+thread count 562 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Jan 04 '24 16:01 hadoop-yetus

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 51s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 1s codespell was not available.
+0 :ok: detsecrets 0m 1s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ trunk Compile Tests _
+1 :green_heart: mvninstall 46m 9s trunk passed
+1 :green_heart: compile 0m 38s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: compile 0m 34s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: checkstyle 0m 30s trunk passed
+1 :green_heart: mvnsite 0m 39s trunk passed
+1 :green_heart: javadoc 0m 37s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javadoc 0m 32s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: spotbugs 1m 4s trunk passed
+1 :green_heart: shadedclient 38m 47s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 0m 29s the patch passed
+1 :green_heart: compile 0m 29s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javac 0m 29s the patch passed
+1 :green_heart: compile 0m 25s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: javac 0m 25s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
-0 :warning: checkstyle 0m 19s /results-checkstyle-hadoop-tools_hadoop-azure.txt hadoop-tools/hadoop-azure: The patch generated 1 new + 6 unchanged - 0 fixed = 7 total (was 6)
+1 :green_heart: mvnsite 0m 29s the patch passed
+1 :green_heart: javadoc 0m 25s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javadoc 0m 24s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: spotbugs 1m 2s the patch passed
+1 :green_heart: shadedclient 37m 46s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: unit 1m 59s hadoop-azure in the patch passed.
+1 :green_heart: asflicense 0m 34s The patch does not generate ASF License warnings.
138m 40s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/2/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6409
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 4853f5102c8b 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / e5605ac3b7beff404cf6147413c3cecac8925b54
Default Java Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/2/testReport/
Max. process+thread count 603 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/2/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Jan 04 '24 17:01 hadoop-yetus

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 57s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 0s codespell was not available.
+0 :ok: detsecrets 0m 0s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ trunk Compile Tests _
+1 :green_heart: mvninstall 41m 39s trunk passed
+1 :green_heart: compile 0m 36s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: compile 0m 33s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: checkstyle 0m 30s trunk passed
+1 :green_heart: mvnsite 0m 38s trunk passed
+1 :green_heart: javadoc 0m 36s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javadoc 0m 31s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: spotbugs 1m 3s trunk passed
+1 :green_heart: shadedclient 32m 33s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 0m 27s the patch passed
+1 :green_heart: compile 0m 28s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javac 0m 28s the patch passed
+1 :green_heart: compile 0m 25s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: javac 0m 25s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
-0 :warning: checkstyle 0m 18s /results-checkstyle-hadoop-tools_hadoop-azure.txt hadoop-tools/hadoop-azure: The patch generated 3 new + 6 unchanged - 0 fixed = 9 total (was 6)
+1 :green_heart: mvnsite 0m 29s the patch passed
+1 :green_heart: javadoc 0m 24s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javadoc 0m 23s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: spotbugs 1m 3s the patch passed
+1 :green_heart: shadedclient 32m 29s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: unit 2m 0s hadoop-azure in the patch passed.
+1 :green_heart: asflicense 0m 34s The patch does not generate ASF License warnings.
122m 22s
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/3/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6409
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux ad3a04c648ae 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 1b0e9a30f9d9016671c0b33dbc61c09c0530b006
Default Java Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/3/testReport/
Max. process+thread count 676 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/3/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Jan 24 '24 12:01 hadoop-yetus

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 59s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 0s codespell was not available.
+0 :ok: detsecrets 0m 0s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ trunk Compile Tests _
+1 :green_heart: mvninstall 43m 8s trunk passed
+1 :green_heart: compile 0m 36s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: compile 0m 33s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: checkstyle 0m 30s trunk passed
+1 :green_heart: mvnsite 0m 38s trunk passed
+1 :green_heart: javadoc 0m 37s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javadoc 0m 33s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: spotbugs 1m 2s trunk passed
+1 :green_heart: shadedclient 32m 17s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 0m 28s the patch passed
+1 :green_heart: compile 0m 28s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javac 0m 28s the patch passed
+1 :green_heart: compile 0m 26s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: javac 0m 26s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
-0 :warning: checkstyle 0m 19s /results-checkstyle-hadoop-tools_hadoop-azure.txt hadoop-tools/hadoop-azure: The patch generated 4 new + 6 unchanged - 0 fixed = 10 total (was 6)
+1 :green_heart: mvnsite 0m 30s the patch passed
+1 :green_heart: javadoc 0m 25s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javadoc 0m 25s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: spotbugs 1m 2s the patch passed
+1 :green_heart: shadedclient 32m 8s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: unit 1m 59s hadoop-azure in the patch passed.
+1 :green_heart: asflicense 0m 32s The patch does not generate ASF License warnings.
123m 26s
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/4/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6409
JIRA Issue HADOOP-18656
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux c984caf97208 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 42010aa466746fe4b6d2ebd7db0d1e6167ca7764
Default Java Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/4/testReport/
Max. process+thread count 551 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/4/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Jan 25 '24 08:01 hadoop-yetus

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 49s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 1s codespell was not available.
+0 :ok: detsecrets 0m 1s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ trunk Compile Tests _
+1 :green_heart: mvninstall 47m 29s trunk passed
+1 :green_heart: compile 0m 38s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: compile 0m 33s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: checkstyle 0m 29s trunk passed
+1 :green_heart: mvnsite 0m 39s trunk passed
+1 :green_heart: javadoc 0m 38s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javadoc 0m 31s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: spotbugs 1m 6s trunk passed
+1 :green_heart: shadedclient 38m 5s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 0m 28s the patch passed
+1 :green_heart: compile 0m 29s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javac 0m 29s the patch passed
+1 :green_heart: compile 0m 25s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: javac 0m 25s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
-0 :warning: checkstyle 0m 19s /results-checkstyle-hadoop-tools_hadoop-azure.txt hadoop-tools/hadoop-azure: The patch generated 1 new + 6 unchanged - 0 fixed = 7 total (was 6)
+1 :green_heart: mvnsite 0m 29s the patch passed
+1 :green_heart: javadoc 0m 25s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javadoc 0m 24s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: spotbugs 1m 4s the patch passed
+1 :green_heart: shadedclient 37m 47s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: unit 1m 58s hadoop-azure in the patch passed.
+1 :green_heart: asflicense 0m 33s The patch does not generate ASF License warnings.
139m 22s
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/5/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6409
JIRA Issue HADOOP-18656
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 29935bf1492c 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / db957389a25f246b4fff0c6b399d56453f338232
Default Java Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/5/testReport/
Max. process+thread count 563 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/5/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Jan 25 '24 08:01 hadoop-yetus

:broken_heart: -1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 35s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 1s codespell was not available.
+0 :ok: detsecrets 0m 1s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ trunk Compile Tests _
-1 :x: mvninstall 50m 25s /branch-mvninstall-root.txt root in trunk failed.
+1 :green_heart: compile 0m 37s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: compile 0m 32s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: checkstyle 0m 31s trunk passed
+1 :green_heart: mvnsite 0m 37s trunk passed
+1 :green_heart: javadoc 0m 36s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javadoc 0m 33s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: spotbugs 1m 3s trunk passed
+1 :green_heart: shadedclient 38m 3s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 0m 27s the patch passed
+1 :green_heart: compile 0m 29s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javac 0m 29s the patch passed
+1 :green_heart: compile 0m 26s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: javac 0m 26s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
-0 :warning: checkstyle 0m 19s /results-checkstyle-hadoop-tools_hadoop-azure.txt hadoop-tools/hadoop-azure: The patch generated 1 new + 6 unchanged - 0 fixed = 7 total (was 6)
+1 :green_heart: mvnsite 0m 29s the patch passed
+1 :green_heart: javadoc 0m 25s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javadoc 0m 25s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: spotbugs 1m 5s the patch passed
+1 :green_heart: shadedclient 38m 51s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: unit 2m 0s hadoop-azure in the patch passed.
+1 :green_heart: asflicense 0m 33s The patch does not generate ASF License warnings.
143m 3s
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/6/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6409
JIRA Issue HADOOP-18656
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 6c13c25f24bd 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 197f3bcbbec9d21c62f063dcee893d7cb1c547ce
Default Java Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/6/testReport/
Max. process+thread count 707 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/6/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Jan 25 '24 09:01 hadoop-yetus

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 58s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 1s codespell was not available.
+0 :ok: detsecrets 0m 1s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ trunk Compile Tests _
+1 :green_heart: mvninstall 41m 30s trunk passed
+1 :green_heart: compile 0m 36s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: compile 0m 33s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: checkstyle 0m 30s trunk passed
+1 :green_heart: mvnsite 0m 39s trunk passed
+1 :green_heart: javadoc 0m 38s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javadoc 0m 33s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: spotbugs 1m 3s trunk passed
+1 :green_heart: shadedclient 32m 8s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 0m 27s the patch passed
+1 :green_heart: compile 0m 28s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javac 0m 28s the patch passed
+1 :green_heart: compile 0m 26s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: javac 0m 26s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
-0 :warning: checkstyle 0m 19s /results-checkstyle-hadoop-tools_hadoop-azure.txt hadoop-tools/hadoop-azure: The patch generated 1 new + 6 unchanged - 0 fixed = 7 total (was 6)
+1 :green_heart: mvnsite 0m 28s the patch passed
+1 :green_heart: javadoc 0m 25s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javadoc 0m 24s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: spotbugs 1m 2s the patch passed
+1 :green_heart: shadedclient 32m 22s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: unit 2m 1s hadoop-azure in the patch passed.
+1 :green_heart: asflicense 0m 35s The patch does not generate ASF License warnings.
121m 59s
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/7/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6409
JIRA Issue HADOOP-18656
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 9339a5fbd571 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 1ab45fa6343269d01eef5a64682eb10b22586ee3
Default Java Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/7/testReport/
Max. process+thread count 702 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/7/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Jan 25 '24 11:01 hadoop-yetus

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 52s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 0s codespell was not available.
+0 :ok: detsecrets 0m 0s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ trunk Compile Tests _
+1 :green_heart: mvninstall 41m 54s trunk passed
+1 :green_heart: compile 0m 38s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: compile 0m 35s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: checkstyle 0m 32s trunk passed
+1 :green_heart: mvnsite 0m 40s trunk passed
+1 :green_heart: javadoc 0m 39s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javadoc 0m 35s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: spotbugs 1m 6s trunk passed
+1 :green_heart: shadedclient 32m 55s branch has no errors when building and testing our client artifacts.
-0 :warning: patch 33m 16s Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 0m 28s the patch passed
+1 :green_heart: compile 0m 30s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javac 0m 30s the patch passed
+1 :green_heart: compile 0m 26s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: javac 0m 27s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
-0 :warning: checkstyle 0m 20s /results-checkstyle-hadoop-tools_hadoop-azure.txt hadoop-tools/hadoop-azure: The patch generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7)
+1 :green_heart: mvnsite 0m 30s the patch passed
+1 :green_heart: javadoc 0m 27s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javadoc 0m 25s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: spotbugs 1m 3s the patch passed
+1 :green_heart: shadedclient 32m 39s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: unit 2m 12s hadoop-azure in the patch passed.
+1 :green_heart: asflicense 0m 37s The patch does not generate ASF License warnings.
124m 26s
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/8/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6409
JIRA Issue HADOOP-18656
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 12c3a833abd7 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 169e1d7e6effb65c77abded66191e50f106c5cdb
Default Java Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/8/testReport/
Max. process+thread count 620 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/8/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Feb 29 '24 13:02 hadoop-yetus

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 53s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 1s No case conflicting files found.
+0 :ok: codespell 0m 0s codespell was not available.
+0 :ok: detsecrets 0m 0s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ trunk Compile Tests _
+1 :green_heart: mvninstall 44m 17s trunk passed
+1 :green_heart: compile 0m 37s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: compile 0m 37s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: checkstyle 0m 32s trunk passed
+1 :green_heart: mvnsite 0m 41s trunk passed
+1 :green_heart: javadoc 0m 39s trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javadoc 0m 35s trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: spotbugs 1m 6s trunk passed
+1 :green_heart: shadedclient 35m 30s branch has no errors when building and testing our client artifacts.
-0 :warning: patch 35m 51s Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 0m 30s the patch passed
+1 :green_heart: compile 0m 30s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javac 0m 30s the patch passed
+1 :green_heart: compile 0m 27s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: javac 0m 27s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
+1 :green_heart: checkstyle 0m 19s the patch passed
+1 :green_heart: mvnsite 0m 30s the patch passed
+1 :green_heart: javadoc 0m 26s the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javadoc 0m 26s the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 :green_heart: spotbugs 1m 3s the patch passed
+1 :green_heart: shadedclient 32m 51s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: unit 2m 11s hadoop-azure in the patch passed.
+1 :green_heart: asflicense 0m 37s The patch does not generate ASF License warnings.
129m 29s
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/9/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6409
JIRA Issue HADOOP-18656
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 0fd962d8422b 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 2b502e4f127af68c0b381f14101b8af0ffb9b72b
Default Java Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/9/testReport/
Max. process+thread count 698 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/9/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Mar 01 '24 10:03 hadoop-yetus

Hi @steveloughran @mukund-thakur Requesting you review on this PR.

Thanks

anujmodi2021 avatar Mar 22 '24 04:03 anujmodi2021

hey, in #6494 i'm drafting a bulk delete API where the caller (iceberg etc) can give a list of file paths for deletion with no guarantees about safety checks, parent dirs existing afterwards etc. Would this work here too?

steveloughran avatar Mar 22 '24 14:03 steveloughran

how does the pagination work here? do repeated calls need to be made? if so, where is this done? it wasn't immediately obvious to me.

Paginated delete will work somehow similar to how recursive delete works for FNS accounts.

For HNS Accounts, recursive delete is supposed to be a O(1) operation i.e deleting the folder itself. But before deleting the whole folder, server needs to do ACL checks on all the children of that folder which is not O(1). If the directory is large, this ACL check can take some time and request can timeout. To avoid this, server will return a continuation token if ACL check is still pending. Client need to loop delete on this continuation token similar to how it loop delete for FNS account recursive delete.

Difference is that for FNS every call does delete some objects, in HNS it only performs ACL checks and actual delete of directory happens on last delete call of loop.

Hope that explains.

Repeated calls are made in abfsStore.deletePath() where continuation token check is present. This is common code for FNS and HNS.

anujmodi2021 avatar Apr 02 '24 10:04 anujmodi2021

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 51s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 0s codespell was not available.
+0 :ok: detsecrets 0m 0s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 6 new or modified test files.
_ trunk Compile Tests _
+1 :green_heart: mvninstall 44m 25s trunk passed
+1 :green_heart: compile 0m 38s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: compile 0m 36s trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 :green_heart: checkstyle 0m 33s trunk passed
+1 :green_heart: mvnsite 0m 41s trunk passed
+1 :green_heart: javadoc 0m 39s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javadoc 0m 36s trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 :green_heart: spotbugs 1m 6s trunk passed
+1 :green_heart: shadedclient 33m 16s branch has no errors when building and testing our client artifacts.
-0 :warning: patch 33m 37s Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 0m 29s the patch passed
+1 :green_heart: compile 0m 29s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javac 0m 29s the patch passed
+1 :green_heart: compile 0m 29s the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 :green_heart: javac 0m 29s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
-0 :warning: checkstyle 0m 20s /results-checkstyle-hadoop-tools_hadoop-azure.txt hadoop-tools/hadoop-azure: The patch generated 1 new + 18 unchanged - 0 fixed = 19 total (was 18)
+1 :green_heart: mvnsite 0m 30s the patch passed
+1 :green_heart: javadoc 0m 26s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javadoc 0m 26s the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 :green_heart: spotbugs 1m 4s the patch passed
+1 :green_heart: shadedclient 33m 19s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: unit 2m 14s hadoop-azure in the patch passed.
+1 :green_heart: asflicense 0m 38s The patch does not generate ASF License warnings.
128m 19s
Subsystem Report/Notes
Docker ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/10/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6409
JIRA Issue HADOOP-18656
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 9a61f5531419 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / d7d9d065dd66ab82a5685f36531b1c91e84265dd
Default Java Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/10/testReport/
Max. process+thread count 730 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/10/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Apr 02 '24 11:04 hadoop-yetus

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 53s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 1s codespell was not available.
+0 :ok: detsecrets 0m 1s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 6 new or modified test files.
_ trunk Compile Tests _
+1 :green_heart: mvninstall 49m 42s trunk passed
+1 :green_heart: compile 0m 42s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: compile 0m 34s trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 :green_heart: checkstyle 0m 31s trunk passed
+1 :green_heart: mvnsite 0m 40s trunk passed
+1 :green_heart: javadoc 0m 39s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javadoc 0m 34s trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 :green_heart: spotbugs 1m 7s trunk passed
+1 :green_heart: shadedclient 39m 59s branch has no errors when building and testing our client artifacts.
-0 :warning: patch 40m 20s Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 0m 29s the patch passed
+1 :green_heart: compile 0m 31s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javac 0m 31s the patch passed
+1 :green_heart: compile 0m 27s the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 :green_heart: javac 0m 27s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
-0 :warning: checkstyle 0m 20s /results-checkstyle-hadoop-tools_hadoop-azure.txt hadoop-tools/hadoop-azure: The patch generated 1 new + 18 unchanged - 0 fixed = 19 total (was 18)
+1 :green_heart: mvnsite 0m 31s the patch passed
+1 :green_heart: javadoc 0m 26s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javadoc 0m 25s the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 :green_heart: spotbugs 1m 7s the patch passed
+1 :green_heart: shadedclient 39m 21s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: unit 2m 12s hadoop-azure in the patch passed.
+1 :green_heart: asflicense 0m 35s The patch does not generate ASF License warnings.
146m 19s
Subsystem Report/Notes
Docker ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/11/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6409
JIRA Issue HADOOP-18656
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux dde2b05887b6 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / f500632b9b690fc139fb724da63bccbca64f5d2f
Default Java Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/11/testReport/
Max. process+thread count 577 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/11/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Apr 02 '24 12:04 hadoop-yetus

hey, in #6494 i'm drafting a bulk delete API where the caller (iceberg etc) can give a list of file paths for deletion with no guarantees about safety checks, parent dirs existing afterwards etc. Would this work here too?

From what I understood from your PR, this seems a bit different from bulk delete. Paginated delete will be supported here but caller won't be able to specify a list of paths that they want to delete. Only one path that too a directory path can be passes and everything inside that directory will be deleted.

Pagination here is only for performing ACL checks and not actual delete. Delete will still be a single operation performed after ACL check is completed on whole directory listing.

In case ACL checks fails in between after a few pages, whole delete operation will fail it won't delete any object. This way this is an atomic delete.

anujmodi2021 avatar Apr 02 '24 12:04 anujmodi2021

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 52s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 1s codespell was not available.
+0 :ok: detsecrets 0m 1s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 6 new or modified test files.
_ trunk Compile Tests _
+1 :green_heart: mvninstall 49m 51s trunk passed
+1 :green_heart: compile 0m 40s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: compile 0m 34s trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 :green_heart: checkstyle 0m 29s trunk passed
+1 :green_heart: mvnsite 0m 39s trunk passed
+1 :green_heart: javadoc 0m 38s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javadoc 0m 35s trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 :green_heart: spotbugs 1m 10s trunk passed
+1 :green_heart: shadedclient 40m 1s branch has no errors when building and testing our client artifacts.
-0 :warning: patch 40m 22s Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 0m 29s the patch passed
+1 :green_heart: compile 0m 30s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javac 0m 30s the patch passed
+1 :green_heart: compile 0m 26s the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 :green_heart: javac 0m 26s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
-0 :warning: checkstyle 0m 20s /results-checkstyle-hadoop-tools_hadoop-azure.txt hadoop-tools/hadoop-azure: The patch generated 1 new + 18 unchanged - 0 fixed = 19 total (was 18)
+1 :green_heart: mvnsite 0m 29s the patch passed
+1 :green_heart: javadoc 0m 28s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javadoc 0m 26s the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 :green_heart: spotbugs 1m 8s the patch passed
+1 :green_heart: shadedclient 39m 13s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: unit 2m 11s hadoop-azure in the patch passed.
+1 :green_heart: asflicense 0m 37s The patch does not generate ASF License warnings.
146m 28s
Subsystem Report/Notes
Docker ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/12/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6409
JIRA Issue HADOOP-18656
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 68fb2c7d0f69 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / a31d3ae0722b103c16f10e6861850245ee64fd40
Default Java Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/12/testReport/
Max. process+thread count 569 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/12/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Apr 02 '24 12:04 hadoop-yetus

:confetti_ball: +1 overall

Vote Subsystem Runtime Logfile Comment
+0 :ok: reexec 0m 54s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+0 :ok: codespell 0m 1s codespell was not available.
+0 :ok: detsecrets 0m 1s detect-secrets was not available.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 6 new or modified test files.
_ trunk Compile Tests _
+1 :green_heart: mvninstall 50m 36s trunk passed
+1 :green_heart: compile 0m 40s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: compile 0m 35s trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 :green_heart: checkstyle 0m 31s trunk passed
+1 :green_heart: mvnsite 0m 39s trunk passed
+1 :green_heart: javadoc 0m 38s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javadoc 0m 35s trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 :green_heart: spotbugs 1m 5s trunk passed
+1 :green_heart: shadedclient 42m 40s branch has no errors when building and testing our client artifacts.
-0 :warning: patch 43m 2s Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 0m 29s the patch passed
+1 :green_heart: compile 0m 30s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javac 0m 30s the patch passed
+1 :green_heart: compile 0m 27s the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 :green_heart: javac 0m 27s the patch passed
+1 :green_heart: blanks 0m 0s The patch has no blanks issues.
+1 :green_heart: checkstyle 0m 20s the patch passed
+1 :green_heart: mvnsite 0m 30s the patch passed
+1 :green_heart: javadoc 0m 27s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 :green_heart: javadoc 0m 24s the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 :green_heart: spotbugs 1m 4s the patch passed
+1 :green_heart: shadedclient 38m 55s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 :green_heart: unit 2m 11s hadoop-azure in the patch passed.
+1 :green_heart: asflicense 0m 36s The patch does not generate ASF License warnings.
149m 42s
Subsystem Report/Notes
Docker ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/13/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/hadoop/pull/6409
JIRA Issue HADOOP-18656
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 78700f0fcb82 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 47f6623fc6c1a0fa8947a231a078b25a1c2befb8
Default Java Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/13/testReport/
Max. process+thread count 528 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6409/13/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus avatar Apr 03 '24 14:04 hadoop-yetus

merged to trunk.

@anujmodi2021 can you do a PR and retest with branch -then I will merge it there too.

steveloughran avatar Apr 04 '24 18:04 steveloughran