hadoop
hadoop copied to clipboard
Published
20 hours ago •
apache
apache
HDFS-17382. Add Apache Log4j Extras Library to Hadoop 3.4 for Enhance…
…d Log Rolling Capabilities
Description of PR
https://issues.apache.org/jira/browse/HDFS-17382
I previously tried to apply branch 3.3 as a target, but I want to merge to branch 3.4 first. link to previous PR
How was this patch tested?
For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
- [ ] If applicable, have you updated the
LICENSE,LICENSE-binary,NOTICE-binaryfiles?
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 0s | Docker mode activated. | |
| -1 :x: | patch | 0m 18s | https://github.com/apache/hadoop/pull/6584 does not apply to branch-3.4. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. |
| Subsystem | Report/Notes |
|---|---|
| GITHUB PR | https://github.com/apache/hadoop/pull/6584 |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6584/3/console |
| versions | git=2.34.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 0s | Docker mode activated. | |
| -1 :x: | patch | 0m 25s | https://github.com/apache/hadoop/pull/6584 does not apply to branch-3.4. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. |
| Subsystem | Report/Notes |
|---|---|
| GITHUB PR | https://github.com/apache/hadoop/pull/6584 |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6584/1/console |
| versions | git=2.34.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 0s | Docker mode activated. | |
| -1 :x: | patch | 0m 14s | https://github.com/apache/hadoop/pull/6584 does not apply to branch-3.4. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. |
| Subsystem | Report/Notes |
|---|---|
| GITHUB PR | https://github.com/apache/hadoop/pull/6584 |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6584/4/console |
| versions | git=2.34.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 0s | Docker mode activated. | |
| -1 :x: | patch | 0m 20s | https://github.com/apache/hadoop/pull/6584 does not apply to branch-3.4. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. |
| Subsystem | Report/Notes |
|---|---|
| GITHUB PR | https://github.com/apache/hadoop/pull/6584 |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6584/2/console |
| versions | git=2.34.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 49s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | xmllint | 0m 0s | xmllint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | |
| _ branch-3.4 Compile Tests _ | ||||
| +0 :ok: | mvndep | 12m 6s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 64m 47s | branch-3.4 passed | |
| +1 :green_heart: | compile | 20m 13s | branch-3.4 passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | compile | 18m 19s | branch-3.4 passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | mvnsite | 2m 53s | branch-3.4 passed | |
| +1 :green_heart: | javadoc | 2m 36s | branch-3.4 passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 3m 5s | branch-3.4 passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | shadedclient | 162m 24s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 37s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 14m 13s | the patch passed | |
| +1 :green_heart: | compile | 19m 8s | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javac | 19m 8s | the patch passed | |
| +1 :green_heart: | compile | 18m 7s | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | javac | 18m 7s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | mvnsite | 2m 49s | the patch passed | |
| +1 :green_heart: | javadoc | 2m 30s | the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 3m 9s | the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08 | |
| +1 :green_heart: | shadedclient | 48m 40s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 0m 31s | hadoop-project in the patch passed. | |
| -1 :x: | unit | 252m 34s | /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 0m 58s | hadoop-client-minicluster in the patch passed. | |
| +1 :green_heart: | asflicense | 1m 11s | The patch does not generate ASF License warnings. | |
| 525m 40s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.protocol.TestBlockListAsLongs |
| hadoop.hdfs.TestRollingUpgrade | |
| hadoop.hdfs.server.datanode.TestLargeBlockReport |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6584/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/6584 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint |
| uname | Linux 650193d11b96 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.4 / 7c60bb5a2d3b3cd91a34be2b0b19c5d49bcfc2f6 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6584/5/testReport/ |
| Max. process+thread count | 2817 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-hdfs-project/hadoop-hdfs hadoop-client-modules/hadoop-client-minicluster U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6584/5/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
@dntjr8096 Thanks for the contribution! @dineshchitlangia Thanks for the review!
I am not sure if we still need to include content from log4j 1.x version. I believe we should not include it.
cc: @ayushtkn @steveloughran
@dineshchitlangia I want to strip dependendencies from hadoop common, rather than add more, as they contaminate all applications downstream and just add more CVEs.
Just as we are adding a hadoop-common-zookeeper, maybe we can add a hadoop-common-server where we add server-side stuff only.
we may still need to stitch these into hadoop/common/lib for release builds, but they won't be exported from the hadoop-common POM as transitive dependencies.
Also, I like to consider 3.3.x as feature complete and we should focus our feature dev on trunk with backports to branch-3.4. non-asf forks can pick what they want...
commented; ignore what I said above about hadoop common, as this is already hadoop hdfs.
however, we are trying to move off log4j 1 entirely. Can't we focus on that?
@steveloughran agreed with moving off log4j1.
@dineshchitlangia @slfan1989 @steveloughran First of all, thank you for reviewing my PR. I also agree with moving away from log4j1. Patching to log4j2 and backporting if possible would be ideal. However, I needed a way to roll 'hdfs-audit.log' with gzip. Since the patch to log4j2 is still in progress, using logrotate was the only way to achieve gzip in the current situation. I believe there are others who would benefit from this feature at the moment. Even if it's a temporary solution, I would appreciate it if you could consider it positively. Thanks.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 00s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 01s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 01s | detect-secrets was not available. | |
| +0 :ok: | xmllint | 0m 01s | xmllint was not available. | |
| +1 :green_heart: | @author | 0m 00s | The patch does not contain any @author tags. | |
| -1 :x: | test4tests | 0m 00s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | |
| _ branch-3.4 Compile Tests _ | ||||
| +0 :ok: | mvndep | 1m 55s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 87m 15s | branch-3.4 passed | |
| +1 :green_heart: | compile | 38m 06s | branch-3.4 passed | |
| +1 :green_heart: | mvnsite | 14m 54s | branch-3.4 passed | |
| +1 :green_heart: | javadoc | 13m 49s | branch-3.4 passed | |
| +1 :green_heart: | shadedclient | 284m 31s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 2m 16s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 22m 44s | the patch passed | |
| +1 :green_heart: | compile | 35m 51s | the patch passed | |
| +1 :green_heart: | javac | 35m 51s | the patch passed | |
| +1 :green_heart: | blanks | 0m 00s | The patch has no blanks issues. | |
| +1 :green_heart: | mvnsite | 14m 44s | the patch passed | |
| +1 :green_heart: | javadoc | 13m 58s | the patch passed | |
| +1 :green_heart: | shadedclient | 163m 36s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 4m 34s | hadoop-project in the patch passed. | |
| -1 :x: | unit | 289m 01s | /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 6m 00s | hadoop-client-minicluster in the patch passed. | |
| -1 :x: | asflicense | 8m 37s | /results-asflicense.txt | The patch generated 2 ASF License warnings. |
| 835m 57s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.cli.TestCacheAdminCLI |
| hadoop.cli.TestXAttrCLI | |
| hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot | |
| hadoop.fs.viewfs.TestViewFileSystemHdfs | |
| hadoop.fs.viewfs.TestViewFileSystemLinkFallback | |
| hadoop.fs.viewfs.TestViewFileSystemLinkMergeSlash | |
| hadoop.fs.viewfs.TestViewFileSystemLinkRegex | |
| hadoop.fs.viewfs.TestViewFileSystemOverloadSchemeHdfsFileSystemContract | |
| hadoop.fs.viewfs.TestViewFsAtHdfsRoot | |
| hadoop.fs.viewfs.TestViewFsHdfs | |
| hadoop.fs.viewfs.TestViewFsLinkFallback | |
| hadoop.hdfs.protocol.TestBlockListAsLongs | |
| hadoop.hdfs.qjournal.server.TestJournalNode | |
| hadoop.hdfs.qjournal.server.TestJournalNodeMXBean | |
| hadoop.hdfs.qjournal.server.TestJournalNodeSync | |
| hadoop.hdfs.server.aliasmap.TestSecureAliasMap | |
| hadoop.hdfs.server.blockmanagement.TestBlockReportLease | |
| hadoop.hdfs.server.blockmanagement.TestBlockTokenWithShortCircuitRead | |
| hadoop.hdfs.server.common.blockaliasmap.impl.TestInMemoryLevelDBAliasMapClient | |
| hadoop.hdfs.server.common.blockaliasmap.impl.TestLevelDBFileRegionAliasMap | |
| hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetCache | |
| hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl | |
| hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList | |
| hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles | |
| hadoop.hdfs.server.datanode.fsdataset.impl.TestProvidedImpl | |
| hadoop.hdfs.server.datanode.TestBlockPoolSliceStorage | |
| hadoop.hdfs.server.datanode.TestBlockScanner | |
| hadoop.hdfs.server.datanode.TestDataNodeLifeline | |
| hadoop.hdfs.server.datanode.TestDataNodeMetrics | |
| hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure | |
| hadoop.hdfs.server.datanode.TestDirectoryScanner | |
| hadoop.hdfs.server.datanode.TestLargeBlockReport | |
| hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand | |
| hadoop.hdfs.server.diskbalancer.TestDiskBalancerRPC | |
| hadoop.hdfs.server.diskbalancer.TestDiskBalancerWithMockMover | |
| hadoop.hdfs.server.diskbalancer.TestPlanner | |
| hadoop.hdfs.server.mover.TestMover | |
| hadoop.hdfs.server.namenode.ha.TestBootstrapAliasmap | |
| hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA | |
| hadoop.hdfs.server.namenode.ha.TestHAAppend | |
| hadoop.hdfs.server.namenode.ha.TestHAMetrics | |
| hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA | |
| hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics | |
| hadoop.hdfs.server.namenode.snapshot.TestOrderedSnapshotDeletionGc | |
| hadoop.hdfs.server.namenode.TestAuditLoggerWithCommands | |
| hadoop.hdfs.server.namenode.TestAuditLogs | |
| hadoop.hdfs.server.namenode.TestCacheDirectives | |
| hadoop.hdfs.server.namenode.TestCacheDirectivesWithViewDFS | |
| hadoop.hdfs.server.namenode.TestEditLog | |
| hadoop.hdfs.server.namenode.TestFsck | |
| hadoop.hdfs.server.namenode.TestFSImage | |
| hadoop.hdfs.server.namenode.TestFSNamesystemLockReport | |
| hadoop.hdfs.server.namenode.TestHDFSConcat | |
| hadoop.hdfs.server.namenode.TestNameNodeMXBean | |
| hadoop.hdfs.server.namenode.TestNameNodeReconfigure | |
| hadoop.hdfs.server.namenode.TestNamenodeRetryCache | |
| hadoop.hdfs.server.namenode.TestNetworkTopologyServlet | |
| hadoop.hdfs.server.namenode.TestNNStorageRetentionManager | |
| hadoop.hdfs.server.namenode.TestProtectedDirectories | |
| hadoop.hdfs.server.namenode.TestReconstructStripedBlocks | |
| hadoop.hdfs.server.namenode.TestSaveNamespace | |
| hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs | |
| hadoop.hdfs.TestDFSShell | |
| hadoop.hdfs.TestDFSUpgradeFromImage | |
| hadoop.hdfs.TestDistributedFileSystem | |
| hadoop.hdfs.TestErasureCodingExerciseAPIs | |
| hadoop.hdfs.TestFetchImage | |
| hadoop.hdfs.TestGetBlocks | |
| hadoop.hdfs.TestQuota | |
| hadoop.hdfs.TestSecureEncryptionZoneWithKMS | |
| hadoop.hdfs.TestTrashWithSecureEncryptionZones | |
| hadoop.hdfs.TestViewDistributedFileSystem | |
| hadoop.hdfs.TestViewDistributedFileSystemContract | |
| hadoop.hdfs.TestViewDistributedFileSystemWithMountLinks | |
| hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer | |
| hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer | |
| hadoop.hdfs.tools.TestDebugAdmin | |
| hadoop.hdfs.tools.TestDFSAdmin | |
| hadoop.hdfs.tools.TestDFSAdminWithHA | |
| hadoop.hdfs.tools.TestViewFileSystemOverloadSchemeWithFSCommands | |
| hadoop.hdfs.tools.TestViewFSStoragePolicyCommands | |
| hadoop.hdfs.web.TestWebHDFS | |
| hadoop.hdfs.web.TestWebHdfsTokens | |
| hadoop.hdfs.web.TestWebHdfsUrl |
| Subsystem | Report/Notes |
|---|---|
| GITHUB PR | https://github.com/apache/hadoop/pull/6584 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint |
| uname | MINGW64_NT-10.0-17763 1d98a7c206b0 3.4.10-87d57229.x86_64 2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | branch-3.4 / 7c60bb5a2d3b3cd91a34be2b0b19c5d49bcfc2f6 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6584/1/testReport/ |
| modules | C: hadoop-project hadoop-hdfs-project/hadoop-hdfs hadoop-client-modules/hadoop-client-minicluster U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6584/1/console |
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
@dntjr8096 Given the larger context as explained by @steveloughran and considering the fact the this is a temporary solution, I am inclined with Steve on this one. May be we could just post a blog and let folks do it locally instead of adding it to the project and then dealing with removal later. Specially the concern about additional CVEs is very valid.