hadoop
hadoop copied to clipboard
apache
HDFS-16703. Enable RPC Timeout for some protocols of NameNode.
Description of PR
When I read some code about protocol, I found that only ClientNamenodeProtocolPB proxy with RPC timeout, other protocolPB proxy without RPC timeout, such as RefreshAuthorizationPolicyProtocolPB, RefreshUserMappingsProtocolPB, RefreshCallQueueProtocolPB, GetUserMappingsProtocolPB and NamenodeProtocolPB.
If proxy without rpc timeout, it will be blocked for a long time if the NN machine crash or bad network during writing or reading with NN.
So I feel that we should enable RPC timeout for all ProtocolPB.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 50s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 14m 50s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 28m 55s | trunk passed | |
| +1 :green_heart: | compile | 6m 49s | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | compile | 6m 20s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | checkstyle | 1m 33s | trunk passed | |
| +1 :green_heart: | mvnsite | 2m 49s | trunk passed | |
| +1 :green_heart: | javadoc | 2m 12s | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | javadoc | 2m 32s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 6m 35s | trunk passed | |
| +1 :green_heart: | shadedclient | 26m 0s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 2m 17s | the patch passed | |
| +1 :green_heart: | compile | 6m 37s | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | javac | 6m 37s | hadoop-hdfs-project-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 0 new + 1134 unchanged - 1 fixed = 1134 total (was 1135) | |
| +1 :green_heart: | compile | 6m 17s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | javac | 6m 17s | hadoop-hdfs-project-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 0 new + 1103 unchanged - 1 fixed = 1103 total (was 1104) | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 1m 16s | the patch passed | |
| +1 :green_heart: | mvnsite | 2m 22s | the patch passed | |
| +1 :green_heart: | javadoc | 1m 42s | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | javadoc | 2m 8s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 6m 20s | the patch passed | |
| +1 :green_heart: | shadedclient | 25m 25s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 2m 31s | hadoop-hdfs-client in the patch passed. | |
| +1 :green_heart: | unit | 339m 19s | hadoop-hdfs in the patch passed. | |
| +1 :green_heart: | asflicense | 1m 3s | The patch does not generate ASF License warnings. | |
| 495m 26s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4660/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4660 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 5ca4b2229ba9 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 03e1c7d8aab9f1f76e5a1ef7fd637fd6808fa991 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4660/1/testReport/ |
| Max. process+thread count | 2282 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4660/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
@ZanderXu I feel that this change is a bit risky, will this lead to instability of the service? Sometimes it is reasonable to configure without timeout. The main question is is it reasonable to use the same timeout configuration for different protocols?
@slfan1989 Thanks for your review.
I feel that this change is a bit risky, will this lead to instability of the service? Sometimes it is reasonable to configure without timeout
Sorry, I didn't get your idea. Can you share more detailed information or cases?
The main question is is it reasonable to use the same timeout configuration for different protocols?
According to my practical experience, using one configuration ipc.client.rpc-timeout.ms is enough. Of course, if some masters feel it is necessary to distinguish this configuration, I will modify this patch.
Sorry, I didn't get your idea. Can you share more detailed information or cases?
I mean you have increased the timeout configuration, will some protocols timeout frequently?
Frequent timeouts cause some operations to fail. I personally think that the performance of NN RPC is limited by NN FsSystemLock, which causes Rpc to not necessarily return at a regular time. Setting this may lead to a large number of RPC failures in some cases, resulting in overall instability.
let's see what other people think.
Thanks @slfan1989 for your explain.
ipc.client.rpc-timeout.ms default is 2min. If client can‘t receive the response within 2min, maybe we should give a high priority to improving the performance of namenode.
If timeout is 0, Client will be blocked for a very long time, such as 15min, due to NN crash or bad network or other reasons.
Thanks @slfan1989 for your explain.
ipc.client.rpc-timeout.msdefault is 2min. If client can‘t receive the response within 2min, maybe we should give a high priority to improving the performance of namenode.If timeout is 0, Client will be blocked for a very long time, such as 15min, due to NN crash or bad network or other reasons.
Thanks for the explanation, from my personal point of view, I don't really agree with this change, but I still listen to other partners' suggestions. I always feel that there is a reason for not setting timeout before.
@goiri Sir, can you help me review this patch? Thanks In our prod environment, we encountered the RBF NameNodeHeartbeatServices thread being blocked for a long time because one of the NameNode machine crashed. So I think this PR will also have an effect on RBF.
I tend to agree that we should avoid not changing the previous behavior. Can we add one setting that allows keeping the old 0 for the default case and tune it for the rest?
@goiri Thanks for your review.
Can we add one setting that allows keeping the old 0 for the default case and tune it for the rest?
Sure, I will modify it with different configuration for different protocol. cc @slfan1989
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 49s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | xmllint | 0m 0s | xmllint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 14m 57s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 28m 11s | trunk passed | |
| +1 :green_heart: | compile | 6m 50s | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | compile | 6m 29s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | checkstyle | 1m 34s | trunk passed | |
| +1 :green_heart: | mvnsite | 2m 49s | trunk passed | |
| +1 :green_heart: | javadoc | 2m 12s | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | javadoc | 2m 29s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 6m 34s | trunk passed | |
| +1 :green_heart: | shadedclient | 25m 47s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 2m 18s | the patch passed | |
| +1 :green_heart: | compile | 6m 39s | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | javac | 6m 39s | hadoop-hdfs-project-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 0 new + 1134 unchanged - 1 fixed = 1134 total (was 1135) | |
| +1 :green_heart: | compile | 6m 15s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | javac | 6m 15s | hadoop-hdfs-project-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 0 new + 1103 unchanged - 1 fixed = 1103 total (was 1104) | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 1m 16s | the patch passed | |
| +1 :green_heart: | mvnsite | 2m 24s | the patch passed | |
| +1 :green_heart: | javadoc | 1m 42s | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | javadoc | 2m 7s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 6m 25s | the patch passed | |
| +1 :green_heart: | shadedclient | 26m 1s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 2m 29s | hadoop-hdfs-client in the patch passed. | |
| +1 :green_heart: | unit | 345m 30s | hadoop-hdfs in the patch passed. | |
| +1 :green_heart: | asflicense | 1m 2s | The patch does not generate ASF License warnings. | |
| 501m 23s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4660/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4660 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux d70449d3d50f 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0f07908a8f36f476647123a5ffab1d6fe187ded6 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4660/2/testReport/ |
| Max. process+thread count | 2481 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4660/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
@ZanderXu Do we need to update the md file to explain what these parameters do?
@slfan1989 Thanks for your review.
Do we need to update the md file to explain what these parameters do?
Do you mean which MD file? ipc.client.rpc-timeout.ms is not been explained. If you think we should do it, I will add some explanation for them.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 41s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | xmllint | 0m 0s | xmllint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 14m 37s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 26m 10s | trunk passed | |
| +1 :green_heart: | compile | 7m 4s | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | compile | 6m 39s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | checkstyle | 1m 31s | trunk passed | |
| +1 :green_heart: | mvnsite | 2m 57s | trunk passed | |
| +1 :green_heart: | javadoc | 2m 14s | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | javadoc | 2m 46s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 6m 53s | trunk passed | |
| +1 :green_heart: | shadedclient | 25m 34s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 32s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 2m 26s | the patch passed | |
| +1 :green_heart: | compile | 6m 45s | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | javac | 6m 45s | hadoop-hdfs-project-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 0 new + 1134 unchanged - 1 fixed = 1134 total (was 1135) | |
| +1 :green_heart: | compile | 6m 34s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | javac | 6m 34s | hadoop-hdfs-project-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 0 new + 1103 unchanged - 1 fixed = 1103 total (was 1104) | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 1m 26s | the patch passed | |
| +1 :green_heart: | mvnsite | 2m 38s | the patch passed | |
| +1 :green_heart: | javadoc | 1m 52s | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | javadoc | 2m 17s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 6m 47s | the patch passed | |
| +1 :green_heart: | shadedclient | 26m 16s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 2m 41s | hadoop-hdfs-client in the patch passed. | |
| +1 :green_heart: | unit | 251m 46s | hadoop-hdfs in the patch passed. | |
| +1 :green_heart: | asflicense | 1m 16s | The patch does not generate ASF License warnings. | |
| 408m 50s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4660/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4660 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux aef4a00016af 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3 10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 07dd3584cd7543a033a6c510787242d977e08750 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4660/3/testReport/ |
| Max. process+thread count | 4031 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4660/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
@slfan1989 Thanks for your review.
Do we need to update the md file to explain what these parameters do?
Do you mean which MD file?
ipc.client.rpc-timeout.msis not been explained. If you think we should do it, I will add some explanation for them.
I'm not sure what is the best place to document. TBH, I do not see a good place to document this; so it might be best to just have it in the XML.
@slfan1989 Master, ping. Could you review it and make some suggestions or approval?
@ZanderXu Sorry for the late reply, the code looks fine. Thank you very much for your contribution.
I am more worried about the use of these timeout configurations, because from my personal point of view, I don't know how to configure it.
Can we provide a guide for configuration?
Such As
ipc.rpc-timeout.for.refresh-user-mappings.ms, if this is not configured to 0, what is a better configuration value?
@slfan1989 Master, thanks for your review.
ipc.rpc-timeout.for.refresh-user-mappings.ms, if this is not configured to 0, what is a better configuration value?
I have added some suggestion values in hdfs-default.xml, such as:
The default value of 0 indicates that timeout is disabled,
which can be set to the same as ipc.client.rpc-timeout.ms, such as 120s.
What do you think of this?
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 53s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | xmllint | 0m 0s | xmllint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 15m 41s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 28m 17s | trunk passed | |
| +1 :green_heart: | compile | 6m 59s | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | compile | 6m 21s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | checkstyle | 1m 34s | trunk passed | |
| +1 :green_heart: | mvnsite | 2m 49s | trunk passed | |
| +1 :green_heart: | javadoc | 2m 12s | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | javadoc | 2m 29s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 6m 35s | trunk passed | |
| +1 :green_heart: | shadedclient | 26m 19s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 2m 19s | the patch passed | |
| +1 :green_heart: | compile | 6m 37s | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | javac | 6m 37s | hadoop-hdfs-project-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 0 new + 1134 unchanged - 1 fixed = 1134 total (was 1135) | |
| +1 :green_heart: | compile | 6m 22s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | javac | 6m 22s | hadoop-hdfs-project-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 0 new + 1103 unchanged - 1 fixed = 1103 total (was 1104) | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 1m 17s | the patch passed | |
| +1 :green_heart: | mvnsite | 2m 23s | the patch passed | |
| +1 :green_heart: | javadoc | 1m 41s | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | javadoc | 2m 7s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 6m 21s | the patch passed | |
| +1 :green_heart: | shadedclient | 25m 59s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 2m 29s | hadoop-hdfs-client in the patch passed. | |
| -1 :x: | unit | 349m 13s | /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 3s | The patch does not generate ASF License warnings. | |
| 506m 47s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.server.mover.TestMover |
| hadoop.hdfs.server.namenode.ha.TestObserverNode |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4660/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4660 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 95644ecbe7f6 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 281d70c8ce50542463a355826d81b07d0ce92fbc |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4660/4/testReport/ |
| Max. process+thread count | 2094 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4660/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
@slfan1989 Sir, can help take a look about this PR? It's helpful for hadoop admin.
@slfan1989 Sir, sorry to ping you again. I still think this is a useful PR for Hadoop Admin. So I hope you can give me some more suggestions and push this PR forward. Thanks again.
:broken_heart: -1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 38s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | xmllint | 0m 0s | xmllint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | |
| _ trunk Compile Tests _ | ||||
| +0 :ok: | mvndep | 16m 24s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 25m 46s | trunk passed | |
| +1 :green_heart: | compile | 6m 6s | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | compile | 5m 55s | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | checkstyle | 1m 37s | trunk passed | |
| +1 :green_heart: | mvnsite | 2m 59s | trunk passed | |
| +1 :green_heart: | javadoc | 2m 14s | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 2m 36s | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 6m 27s | trunk passed | |
| +1 :green_heart: | shadedclient | 23m 4s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 32s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 2m 14s | the patch passed | |
| +1 :green_heart: | compile | 6m 1s | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javac | 6m 1s | hadoop-hdfs-project-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 generated 0 new + 1134 unchanged - 1 fixed = 1134 total (was 1135) | |
| +1 :green_heart: | compile | 5m 41s | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | javac | 5m 41s | hadoop-hdfs-project-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 generated 0 new + 1103 unchanged - 1 fixed = 1103 total (was 1104) | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 1m 16s | the patch passed | |
| +1 :green_heart: | mvnsite | 2m 23s | the patch passed | |
| +1 :green_heart: | javadoc | 1m 39s | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 2m 8s | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | spotbugs | 6m 1s | the patch passed | |
| +1 :green_heart: | shadedclient | 22m 37s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 2m 38s | hadoop-hdfs-client in the patch passed. | |
| +1 :green_heart: | unit | 239m 8s | hadoop-hdfs in the patch passed. | |
| +1 :green_heart: | asflicense | 1m 8s | The patch does not generate ASF License warnings. | |
| 385m 59s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4660/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4660 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux ae65e76df820 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9967ad85b45410dd431391942dd7faad1eedbad0 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4660/5/testReport/ |
| Max. process+thread count | 3798 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4660/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
@slfan1989 Sir, sorry to ping you again. I'm looking forward to your feedback. RPC Timeout is needed to prevent long-term blocking when the network is bad or machine crash. Thanks again.
@slfan1989 Sir, about this PR, if you thinks it's difficult to configure it, how about just enabling a configurable timeout for NamenodeProtocolPB?
Because we encountered many times this problem in our prod environment that RBF can not sense the crashed namenode in time, because the NamenodeHeartbeatService are blocked to waiting for the response from namenode for a long time.