hadoop
hadoop copied to clipboard
Published
20 hours ago •
apache
apache
HADOOP-18476. Abfs and S3A FileContext bindings to close wrapped filesystems in finalizer
Not a fan of finalize, especially as FileContext delete on exit seems trouble. The class holds strong references so finalize() won't be called, ever.
Ignoring that, and avoiding something major like making everything closeable, the abfs and s3a bindings should close their wrapped FS in close.
How was this patch tested?
s3a tests: london, scale abfs: cardiff, scale
For code changes:
- [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [X] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
- [ ] If applicable, have you updated the
LICENSE,LICENSE-binary,NOTICE-binaryfiles?
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 1m 5s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 1s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 1s | detect-secrets was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 15m 33s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 28m 45s | trunk passed | |
| +1 :green_heart: | compile | 3m 21s | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | compile | 2m 32s | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | checkstyle | 0m 56s | trunk passed | |
| +1 :green_heart: | mvnsite | 1m 35s | trunk passed | |
| +1 :green_heart: | javadoc | 1m 10s | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 1m 12s | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 2m 31s | trunk passed | |
| +1 :green_heart: | shadedclient | 23m 23s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 1m 9s | the patch passed | |
| +1 :green_heart: | compile | 3m 10s | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 | |
| -1 :x: | javac | 3m 10s | /results-compile-javac-hadoop-tools-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04.txt | hadoop-tools-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 generated 6 new + 336 unchanged - 0 fixed = 342 total (was 336) |
| +1 :green_heart: | compile | 2m 20s | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | javac | 2m 20s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 0m 43s | /results-checkstyle-hadoop-tools.txt | hadoop-tools: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | mvnsite | 1m 14s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 47s | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 0m 54s | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 2m 21s | the patch passed | |
| +1 :green_heart: | shadedclient | 23m 19s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 2m 44s | hadoop-aws in the patch passed. | |
| +1 :green_heart: | unit | 2m 5s | hadoop-azure in the patch passed. | |
| +1 :green_heart: | asflicense | 0m 41s | The patch does not generate ASF License warnings. | |
| 125m 59s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4966/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4966 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux f2fb462bd959 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fd1aced2d3b7feb4e303bfd2023e735eb95d075c |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4966/1/testReport/ |
| Max. process+thread count | 530 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: hadoop-tools |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4966/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
he class holds strong references so finalize() won't be called, ever.
Are you referring to DELETE_ON_EXIT map?
Changes looks good. there is just one checkstyle reported.
@mukund-thakur yes. once someone registers a deleteOnExit file, a strong ref to the FC is retained along with the list of files to delete. compare with the FS version, which is really "delete when close() is called"
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 1m 22s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 1s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 1s | detect-secrets was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 16m 31s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 32m 12s | trunk passed | |
| +1 :green_heart: | compile | 3m 52s | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | compile | 2m 58s | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | checkstyle | 1m 9s | trunk passed | |
| +1 :green_heart: | mvnsite | 2m 3s | trunk passed | |
| +1 :green_heart: | javadoc | 1m 25s | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 1m 28s | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 2m 56s | trunk passed | |
| +1 :green_heart: | shadedclient | 27m 13s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 1m 17s | the patch passed | |
| +1 :green_heart: | compile | 3m 25s | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 | |
| -1 :x: | javac | 3m 25s | /results-compile-javac-hadoop-tools-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04.txt | hadoop-tools-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 generated 6 new + 326 unchanged - 0 fixed = 332 total (was 326) |
| +1 :green_heart: | compile | 2m 43s | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | javac | 2m 43s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 0m 44s | the patch passed | |
| +1 :green_heart: | mvnsite | 1m 27s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 57s | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 1m 8s | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 2m 45s | the patch passed | |
| +1 :green_heart: | shadedclient | 26m 53s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 3m 3s | hadoop-aws in the patch passed. | |
| +1 :green_heart: | unit | 2m 18s | hadoop-azure in the patch passed. | |
| +1 :green_heart: | asflicense | 0m 52s | The patch does not generate ASF License warnings. | |
| 143m 13s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4966/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4966 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 8faa1f5d0681 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6eaf6fa1e150e031d83f25fd0452eff8d12b2fc7 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4966/2/testReport/ |
| Max. process+thread count | 613 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure U: hadoop-tools |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4966/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
you know, the other thing to think of here is have finalize() methods in s3a/abfs, though they aren't ever encouraged
you know, the other thing to think of here is have finalize() methods in s3a/abfs, though they aren't ever encouraged
You mean in the S3AFileSystem and ABFS right? Yes, I guess we can add but then there are close() methods already there.