hadoop
hadoop copied to clipboard
apache
YARN-11112 Avoid renewing delegation token when app is first submitte…
…d to RM
Description of PR
When auth is enabled by NameNode, then delegation token is required if application needs to acess files/directoies. We find that when app is first submitted to RM, RM renewer will renew app token no matter whether token is expired or not. Renewing token is a bit heavy since it uses global write lock. Here is the result when delegation token is required in a very busy cluster.
How was this patch tested?
For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
- [ ] If applicable, have you updated the
LICENSE,LICENSE-binary,NOTICE-binaryfiles?
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 39s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 38m 51s | trunk passed | |
| +1 :green_heart: | compile | 1m 20s | trunk passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | |
| +1 :green_heart: | compile | 1m 15s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | checkstyle | 1m 10s | trunk passed | |
| +1 :green_heart: | mvnsite | 1m 20s | trunk passed | |
| +1 :green_heart: | javadoc | 1m 13s | trunk passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | |
| +1 :green_heart: | javadoc | 1m 2s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 2m 30s | trunk passed | |
| +1 :green_heart: | shadedclient | 22m 7s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 0m 57s | the patch passed | |
| +1 :green_heart: | compile | 1m 5s | the patch passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | |
| +1 :green_heart: | javac | 1m 5s | the patch passed | |
| +1 :green_heart: | compile | 0m 57s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | javac | 0m 57s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 0m 48s | /results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 2 new + 78 unchanged - 0 fixed = 80 total (was 78) |
| +1 :green_heart: | mvnsite | 1m 2s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 48s | the patch passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | |
| +1 :green_heart: | javadoc | 0m 43s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 2m 5s | the patch passed | |
| +1 :green_heart: | shadedclient | 22m 53s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| -1 :x: | unit | 54m 33s | /patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | hadoop-yarn-server-resourcemanager in the patch passed. |
| +0 :ok: | asflicense | 0m 39s | ASF License check generated no output? | |
| 157m 5s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation |
| hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation | |
| hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4198/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4198 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 0b2a9ca83423 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d9c40fc24236058435a5e4f59f2be0a0646a1456 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4198/1/testReport/ |
| Max. process+thread count | 692 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4198/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 40s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 1s | codespell was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| -1 :x: | mvninstall | 39m 36s | /branch-mvninstall-root.txt | root in trunk failed. |
| +1 :green_heart: | compile | 1m 24s | trunk passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | |
| +1 :green_heart: | compile | 1m 17s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | checkstyle | 1m 9s | trunk passed | |
| +1 :green_heart: | mvnsite | 1m 23s | trunk passed | |
| +1 :green_heart: | javadoc | 1m 13s | trunk passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | |
| +1 :green_heart: | javadoc | 1m 1s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 2m 32s | trunk passed | |
| +1 :green_heart: | shadedclient | 22m 24s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 1m 0s | the patch passed | |
| +1 :green_heart: | compile | 1m 4s | the patch passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | |
| +1 :green_heart: | javac | 1m 4s | the patch passed | |
| +1 :green_heart: | compile | 0m 57s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | javac | 0m 57s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 0m 48s | the patch passed | |
| +1 :green_heart: | mvnsite | 1m 1s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 47s | the patch passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | |
| +1 :green_heart: | javadoc | 0m 45s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 2m 1s | the patch passed | |
| +1 :green_heart: | shadedclient | 21m 7s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| -1 :x: | unit | 98m 10s | /patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 50s | The patch does not generate ASF License warnings. | |
| 200m 28s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4198/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4198 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 992a30706cfa 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d753feffc5e0c65dd521c69987547c43b960228a |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4198/2/testReport/ |
| Max. process+thread count | 951 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4198/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
@steveloughran Thanks for your comment.
Let's say a user submit an app,
- submit by client (2022-04-20:20:00)
- generate delegation token(2022-04-20:20:01)
- then the delegation token expired time is (2022-04-27:20:00)
- RM get the submitting event, process it asynchronously.
- RM may be busy or not, suppose that the app is handled after several minutes(2022-04-20:25:01)
- without this patch, RM will renew the token when handling submitting event, then the expire time is (2022-04-27 20:25:01)
- next time for renewing this token is 2022-04-27 08:13:00 = (2022-04-27 20:25:01) - (7 * 24 * 3600 * 1000) / 10
The Step 6 is not very necesarry as the expire time does not vary a lot (only several minutes) . I'll go through the history to see why RM needs to renew token immediately.
Renewing token will be a heavy opertion in a very busy cluster(in our case, there are nearly 40, 000 apps an hour), as it uses namesystem write lock.
--> And we can't assume that every hosts clocks in perfect sync Renewing immediately is not very helpful in this case, so I think it's harmless to remove Step 6.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 1m 30s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 1s | codespell was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 39m 23s | trunk passed | |
| +1 :green_heart: | compile | 1m 18s | trunk passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | |
| +1 :green_heart: | compile | 1m 16s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | checkstyle | 1m 9s | trunk passed | |
| +1 :green_heart: | mvnsite | 1m 19s | trunk passed | |
| +1 :green_heart: | javadoc | 1m 13s | trunk passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | |
| +1 :green_heart: | javadoc | 1m 2s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 2m 35s | trunk passed | |
| +1 :green_heart: | shadedclient | 21m 34s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 0m 58s | the patch passed | |
| +1 :green_heart: | compile | 1m 5s | the patch passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | |
| +1 :green_heart: | javac | 1m 5s | the patch passed | |
| +1 :green_heart: | compile | 0m 59s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | javac | 0m 59s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 0m 46s | /results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 2 new + 75 unchanged - 3 fixed = 77 total (was 78) |
| +1 :green_heart: | mvnsite | 1m 6s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 50s | the patch passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | |
| +1 :green_heart: | javadoc | 0m 48s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 2m 13s | the patch passed | |
| +1 :green_heart: | shadedclient | 21m 16s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 98m 59s | hadoop-yarn-server-resourcemanager in the patch passed. | |
| +1 :green_heart: | asflicense | 0m 54s | The patch does not generate ASF License warnings. | |
| 201m 4s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4198/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4198 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux ef07b8ec0a3f 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c22cbe8b1db63a94cf3c51a63379b55bcb0ecccf |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4198/3/testReport/ |
| Max. process+thread count | 942 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4198/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 39s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 39m 30s | trunk passed | |
| +1 :green_heart: | compile | 1m 23s | trunk passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | |
| +1 :green_heart: | compile | 1m 13s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | checkstyle | 1m 3s | trunk passed | |
| +1 :green_heart: | mvnsite | 1m 22s | trunk passed | |
| +1 :green_heart: | javadoc | 1m 9s | trunk passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | |
| +1 :green_heart: | javadoc | 0m 56s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 2m 29s | trunk passed | |
| +1 :green_heart: | shadedclient | 21m 58s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 1m 0s | the patch passed | |
| +1 :green_heart: | compile | 1m 6s | the patch passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | |
| +1 :green_heart: | javac | 1m 6s | the patch passed | |
| +1 :green_heart: | compile | 1m 1s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | javac | 1m 1s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 0m 46s | /results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 2 new + 75 unchanged - 3 fixed = 77 total (was 78) |
| +1 :green_heart: | mvnsite | 1m 2s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 47s | the patch passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | |
| +1 :green_heart: | javadoc | 0m 42s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 2m 8s | the patch passed | |
| +1 :green_heart: | shadedclient | 21m 14s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 98m 34s | hadoop-yarn-server-resourcemanager in the patch passed. | |
| +1 :green_heart: | asflicense | 0m 48s | The patch does not generate ASF License warnings. | |
| 200m 1s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4198/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4198 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux c645ece13e53 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c22cbe8b1db63a94cf3c51a63379b55bcb0ecccf |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4198/4/testReport/ |
| Max. process+thread count | 949 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4198/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
@dineshchitlangia @9uapaw @brumi1024 If u have time, could u help review it? I think it's a nice improvement.
@zuston thanks for your comment. We're also working on HDFS-16558 to change the write lock of delegation token to read lock
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 38s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 39m 8s | trunk passed | |
| +1 :green_heart: | compile | 1m 20s | trunk passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | |
| +1 :green_heart: | compile | 1m 15s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | checkstyle | 1m 12s | trunk passed | |
| +1 :green_heart: | mvnsite | 1m 19s | trunk passed | |
| +1 :green_heart: | javadoc | 1m 13s | trunk passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | |
| +1 :green_heart: | javadoc | 1m 3s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 2m 24s | trunk passed | |
| +1 :green_heart: | shadedclient | 21m 49s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 0m 58s | the patch passed | |
| +1 :green_heart: | compile | 1m 5s | the patch passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | |
| +1 :green_heart: | javac | 1m 5s | the patch passed | |
| +1 :green_heart: | compile | 0m 58s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | javac | 0m 58s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 0m 48s | /results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 75 unchanged - 3 fixed = 76 total (was 78) |
| +1 :green_heart: | mvnsite | 1m 1s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 47s | the patch passed with JDK Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 | |
| +1 :green_heart: | javadoc | 0m 46s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 2m 5s | the patch passed | |
| +1 :green_heart: | shadedclient | 21m 28s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 97m 43s | hadoop-yarn-server-resourcemanager in the patch passed. | |
| +1 :green_heart: | asflicense | 0m 48s | The patch does not generate ASF License warnings. | |
| 199m 12s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4198/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4198 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 511bcdbc2937 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ece37d683b383236d8147e673b63cddf72f903c7 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4198/5/testReport/ |
| Max. process+thread count | 967 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4198/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 46s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 1s | codespell was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 14m 41s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 26m 9s | trunk passed | |
| +1 :green_heart: | compile | 10m 41s | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | compile | 9m 24s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | checkstyle | 2m 8s | trunk passed | |
| +1 :green_heart: | mvnsite | 3m 1s | trunk passed | |
| +1 :green_heart: | javadoc | 2m 49s | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | javadoc | 2m 45s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 5m 39s | trunk passed | |
| +1 :green_heart: | shadedclient | 22m 52s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch | |
| -1 :x: | mvninstall | 0m 26s | /patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | compile | 3m 28s | /patch-compile-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt | hadoop-yarn in the patch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | javac | 3m 28s | /patch-compile-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt | hadoop-yarn in the patch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | compile | 3m 0s | /patch-compile-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt | hadoop-yarn in the patch failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| -1 :x: | javac | 3m 0s | /patch-compile-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt | hadoop-yarn in the patch failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 1m 0s | /buildtool-patch-checkstyle-hadoop-yarn-project_hadoop-yarn.txt | The patch fails to run checkstyle in hadoop-yarn |
| -1 :x: | mvnsite | 0m 32s | /patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | javadoc | 0m 33s | /patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt | hadoop-yarn-server-resourcemanager in the patch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | javadoc | 0m 31s | /patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt | hadoop-yarn-server-resourcemanager in the patch failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| -1 :x: | spotbugs | 0m 30s | /patch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | shadedclient | 10m 0s | patch has errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| -1 :x: | unit | 1m 8s | /patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt | hadoop-yarn-api in the patch passed. |
| -1 :x: | unit | 0m 31s | /patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | hadoop-yarn-server-resourcemanager in the patch failed. |
| +1 :green_heart: | asflicense | 0m 42s | The patch does not generate ASF License warnings. | |
| 128m 19s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.yarn.conf.TestYarnConfigurationFields |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4198/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4198 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux f723591143bb 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / edc1db86c7943e34773de58b97bbb9c4c7ab504d |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4198/6/testReport/ |
| Max. process+thread count | 677 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4198/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 41s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 14m 41s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 24m 58s | trunk passed | |
| +1 :green_heart: | compile | 10m 3s | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | compile | 9m 2s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | checkstyle | 2m 20s | trunk passed | |
| +1 :green_heart: | mvnsite | 3m 21s | trunk passed | |
| +1 :green_heart: | javadoc | 3m 6s | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | javadoc | 2m 52s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 5m 21s | trunk passed | |
| +1 :green_heart: | shadedclient | 22m 47s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 1m 45s | the patch passed | |
| +1 :green_heart: | compile | 9m 7s | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | javac | 9m 7s | the patch passed | |
| +1 :green_heart: | compile | 8m 51s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | javac | 8m 51s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 2m 6s | /results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt | hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 239 unchanged - 3 fixed = 240 total (was 242) |
| +1 :green_heart: | mvnsite | 3m 1s | the patch passed | |
| +1 :green_heart: | javadoc | 2m 36s | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | javadoc | 2m 31s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 5m 14s | the patch passed | |
| +1 :green_heart: | shadedclient | 23m 41s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| -1 :x: | unit | 1m 38s | /patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt | hadoop-yarn-api in the patch passed. |
| -1 :x: | unit | 98m 19s | /patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 1m 25s | The patch does not generate ASF License warnings. | |
| 262m 19s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.yarn.conf.TestYarnConfigurationFields |
| hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4198/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4198 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 3bbf13f06a6c 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2687aff21abf6fbc4a3853d5fb4d8867a63ed759 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4198/8/testReport/ |
| Max. process+thread count | 955 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4198/8/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 37s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 1s | codespell was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 14m 41s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 26m 6s | trunk passed | |
| +1 :green_heart: | compile | 10m 46s | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | compile | 9m 27s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | checkstyle | 2m 24s | trunk passed | |
| +1 :green_heart: | mvnsite | 3m 10s | trunk passed | |
| +1 :green_heart: | javadoc | 3m 2s | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | javadoc | 2m 39s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 5m 38s | trunk passed | |
| +1 :green_heart: | shadedclient | 23m 2s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 32s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 1m 42s | the patch passed | |
| +1 :green_heart: | compile | 10m 8s | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | javac | 10m 8s | the patch passed | |
| +1 :green_heart: | compile | 9m 2s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | javac | 9m 2s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 2m 6s | /results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt | hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 239 unchanged - 3 fixed = 240 total (was 242) |
| +1 :green_heart: | mvnsite | 2m 48s | the patch passed | |
| +1 :green_heart: | javadoc | 2m 37s | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | javadoc | 2m 30s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 5m 16s | the patch passed | |
| +1 :green_heart: | shadedclient | 23m 2s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| -1 :x: | unit | 1m 41s | /patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt | hadoop-yarn-api in the patch passed. |
| -1 :x: | unit | 99m 1s | /patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 1m 15s | The patch does not generate ASF License warnings. | |
| 265m 26s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.yarn.conf.TestYarnConfigurationFields |
| hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4198/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4198 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux d04aeff44ebf 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2687aff21abf6fbc4a3853d5fb4d8867a63ed759 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4198/7/testReport/ |
| Max. process+thread count | 927 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4198/7/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 38s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 14m 27s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 25m 41s | trunk passed | |
| +1 :green_heart: | compile | 10m 49s | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | compile | 9m 56s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | checkstyle | 2m 21s | trunk passed | |
| +1 :green_heart: | mvnsite | 4m 22s | trunk passed | |
| +1 :green_heart: | javadoc | 4m 9s | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | javadoc | 3m 55s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 7m 46s | trunk passed | |
| +1 :green_heart: | shadedclient | 23m 53s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 2m 33s | the patch passed | |
| +1 :green_heart: | compile | 9m 9s | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | javac | 9m 9s | the patch passed | |
| +1 :green_heart: | compile | 8m 44s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | javac | 8m 44s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| -0 :warning: | checkstyle | 2m 4s | /results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt | hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 239 unchanged - 3 fixed = 240 total (was 242) |
| +1 :green_heart: | mvnsite | 4m 19s | the patch passed | |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | |
| +1 :green_heart: | javadoc | 3m 57s | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | javadoc | 3m 48s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 7m 39s | the patch passed | |
| +1 :green_heart: | shadedclient | 23m 12s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 1m 41s | hadoop-yarn-api in the patch passed. | |
| +1 :green_heart: | unit | 5m 31s | hadoop-yarn-common in the patch passed. | |
| +1 :green_heart: | unit | 98m 47s | hadoop-yarn-server-resourcemanager in the patch passed. | |
| +1 :green_heart: | asflicense | 1m 25s | The patch does not generate ASF License warnings. | |
| 283m 50s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4198/9/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4198 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux 5ce04fc1a62d 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fba0d3bbb406b28bfac2812f678d8567d92bb0da |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4198/9/testReport/ |
| Max. process+thread count | 949 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4198/9/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
For those who is interesting in this patch, we've applied it in our product env, and it works as expected.
Thanks for the patch @yuanboliu. I was not involved in the implementation of this class, however, according to: https://blog.cloudera.com/hadoop-delegation-tokens-explained/ the immediate renewal is necessary due to "YARN RM verifies the Delegation Tokens are valid by immediately renewing them." What is you opinion about it @steveloughran @Hexiaoqiao?
@9uapaw @Hexiaoqiao @yuanboliu I would like to ask, RM_DELEGATIONTOKEN, this TOKEN is generated by RM, but I don't see the logic to verify this TOKEN，can you give some advice？
Sorry for the late reply 9uapaw Using renewing to verify the validation of token is not very convincible, the invaild token will be rejected as soon as job start to read files from hdfs.
@slfan1989 Basically this thread is used for jobs to renew token from third-party service. RM_DELEGATIONTOKEN is generated from RM and it should be renewed from the client, not the server.
As for the verification, you can refer it to RMDelegationTokenSecretManager which extends from AbstractDelegationTokenSecretManager checkToken.
For those who's interesting in this patch: We provide this patch, because our production clusters are very large numbers, accessing data cross clusters is required. So verifying tokens becomes a heavy rpc. I'm not very insist on merging this patch into trunk as the origin verification is very fast(nearly 1ms).
We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you feel like this was a mistake, or you would like to continue working on it, please feel free to re-open it and ask for a committer to remove the stale tag and review again. Thanks all for your contribution.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}