hadoop
hadoop copied to clipboard
HDFS-16695. Improve code with Lambda in org.apache.hadoop.hdfs.servernamenode package
Description of PR
Improve Code with Lambda in org.apahce.hadoop.hdfs.server.namenode package.
For example: Current logic:
public ListenableFuture<GetJournaledEditsResponseProto> getJournaledEdits(
long fromTxnId, int maxTransactions) {
return parallelExecutor.submit(
new Callable<GetJournaledEditsResponseProto>() {
@Override
public GetJournaledEditsResponseProto call() throws IOException {
return getProxy().getJournaledEdits(journalId, nameServiceId,
fromTxnId, maxTransactions);
}
});
}
Improved Code with Lambda:
public ListenableFuture<GetJournaledEditsResponseProto> getJournaledEdits(
long fromTxnId, int maxTransactions) {
return parallelExecutor.submit(() -> getProxy().getJournaledEdits(
journalId, nameServiceId, fromTxnId, maxTransactions));
}
This is a massive PR. I'm not sure if it is worth though. I would prefer to scope it too.
This is a massive PR. I'm not sure if it is worth though. I would prefer to scope it too.
Personally, I think this optimization should still be valuable, I hope that the submitted pr is not at the class level, at least at the moudle level.
Personally, I think this optimization should still be valuable, I hope that the submitted pr is not at the class level, at least at the moudle level.
I wouldn't go for one per class but maybe subpackage. I also think this PR is doing a couple of things more other than adding lambdas. For example, adjusting for iterations.
:broken_heart: -1 overall
Vote | Subsystem | Runtime | Logfile | Comment |
---|---|---|---|---|
+0 :ok: | reexec | 0m 38s | Docker mode activated. | |
_ Prechecks _ | ||||
+1 :green_heart: | dupname | 0m 2s | No case conflicting files found. | |
+0 :ok: | codespell | 0m 0s | codespell was not available. | |
+0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
+1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
-1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | |
_ trunk Compile Tests _ | ||||
+1 :green_heart: | mvninstall | 38m 27s | trunk passed | |
+1 :green_heart: | compile | 1m 44s | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
+1 :green_heart: | compile | 1m 36s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
+1 :green_heart: | checkstyle | 1m 34s | trunk passed | |
+1 :green_heart: | mvnsite | 1m 46s | trunk passed | |
+1 :green_heart: | javadoc | 1m 23s | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
+1 :green_heart: | javadoc | 1m 52s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
+1 :green_heart: | spotbugs | 3m 44s | trunk passed | |
+1 :green_heart: | shadedclient | 23m 15s | branch has no errors when building and testing our client artifacts. | |
_ Patch Compile Tests _ | ||||
+1 :green_heart: | mvninstall | 1m 24s | the patch passed | |
+1 :green_heart: | compile | 1m 26s | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
+1 :green_heart: | javac | 1m 26s | the patch passed | |
+1 :green_heart: | compile | 1m 21s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
+1 :green_heart: | javac | 1m 21s | the patch passed | |
+1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
-0 :warning: | checkstyle | 1m 10s | /results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | hadoop-hdfs-project/hadoop-hdfs: The patch generated 17 new + 1047 unchanged - 59 fixed = 1064 total (was 1106) |
+1 :green_heart: | mvnsite | 1m 25s | the patch passed | |
+1 :green_heart: | javadoc | 0m 57s | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
+1 :green_heart: | javadoc | 1m 30s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
-1 :x: | spotbugs | 3m 25s | /new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html | hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
+1 :green_heart: | shadedclient | 22m 45s | patch has no errors when building and testing our client artifacts. | |
_ Other Tests _ | ||||
+1 :green_heart: | unit | 240m 48s | hadoop-hdfs in the patch passed. | |
+1 :green_heart: | asflicense | 1m 13s | The patch does not generate ASF License warnings. | |
351m 53s |
Reason | Tests |
---|---|
SpotBugs | module:hadoop-hdfs-project/hadoop-hdfs |
Dead store to size in org.apache.hadoop.hdfs.server.namenode.INodeWithAdditionalFields.getFeature(Class) At INodeWithAdditionalFields.java:org.apache.hadoop.hdfs.server.namenode.INodeWithAdditionalFields.getFeature(Class) At INodeWithAdditionalFields.java:[line 347] |
Subsystem | Report/Notes |
---|---|
Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4668/1/artifact/out/Dockerfile |
GITHUB PR | https://github.com/apache/hadoop/pull/4668 |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
uname | Linux 4d1833debfc7 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | dev-support/bin/hadoop.sh |
git revision | trunk / 2f84f33c37de457c1563a7fe6e24636bac750b8d |
Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4668/1/testReport/ |
Max. process+thread count | 3064 (vs. ulimit of 5500) |
modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4668/1/console |
versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
Personally, I think this optimization should still be valuable, I hope that the submitted pr is not at the class level, at least at the moudle level.
I wouldn't go for one per class but maybe subpackage. I also think this PR is doing a couple of things more other than adding lambdas. For example, adjusting for iterations.
I understand your suggestion that the lambda should be tuned with other code modifications, not just the lambda code.
Thanks @goiri @slfan1989 for your review.
- First, I will remove some code that has nothing to do with lambdas
- Second, I will split this PR at the subPackage Level, such as:
org.apache.hadoop.hdfs.server.namenode
,org.apache.hadoop.hdfs.server.namenode.ha
@goiri @slfan1989 Sir, I have updated this patch, please help me review it. Thanks
- Just improved code with Lambda in namenode package
- Removed some code that has nothing to do with lambdas
:broken_heart: -1 overall
Vote | Subsystem | Runtime | Logfile | Comment |
---|---|---|---|---|
+0 :ok: | reexec | 0m 51s | Docker mode activated. | |
_ Prechecks _ | ||||
+1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | |
+0 :ok: | codespell | 0m 0s | codespell was not available. | |
+0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
+1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
-1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | |
_ trunk Compile Tests _ | ||||
+1 :green_heart: | mvninstall | 41m 19s | trunk passed | |
+1 :green_heart: | compile | 1m 41s | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
+1 :green_heart: | compile | 1m 30s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
+1 :green_heart: | checkstyle | 1m 24s | trunk passed | |
+1 :green_heart: | mvnsite | 1m 40s | trunk passed | |
+1 :green_heart: | javadoc | 1m 20s | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
+1 :green_heart: | javadoc | 1m 39s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
+1 :green_heart: | spotbugs | 3m 45s | trunk passed | |
+1 :green_heart: | shadedclient | 26m 7s | branch has no errors when building and testing our client artifacts. | |
_ Patch Compile Tests _ | ||||
+1 :green_heart: | mvninstall | 1m 23s | the patch passed | |
+1 :green_heart: | compile | 1m 29s | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
+1 :green_heart: | javac | 1m 29s | the patch passed | |
+1 :green_heart: | compile | 1m 18s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
+1 :green_heart: | javac | 1m 18s | the patch passed | |
+1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
-0 :warning: | checkstyle | 1m 5s | /results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 777 unchanged - 39 fixed = 781 total (was 816) |
+1 :green_heart: | mvnsite | 1m 25s | the patch passed | |
+1 :green_heart: | javadoc | 1m 0s | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
+1 :green_heart: | javadoc | 1m 29s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
+1 :green_heart: | spotbugs | 3m 31s | the patch passed | |
+1 :green_heart: | shadedclient | 25m 38s | patch has no errors when building and testing our client artifacts. | |
_ Other Tests _ | ||||
+1 :green_heart: | unit | 336m 46s | hadoop-hdfs in the patch passed. | |
+1 :green_heart: | asflicense | 1m 0s | The patch does not generate ASF License warnings. | |
454m 44s |
Subsystem | Report/Notes |
---|---|
Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4668/2/artifact/out/Dockerfile |
GITHUB PR | https://github.com/apache/hadoop/pull/4668 |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
uname | Linux ceeae7be4049 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | dev-support/bin/hadoop.sh |
git revision | trunk / e5ea21a2a5c553951c9b1dfc2a25bcabc68bcac3 |
Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4668/2/testReport/ |
Max. process+thread count | 1846 (vs. ulimit of 5500) |
modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4668/2/console |
versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
:broken_heart: -1 overall
Vote | Subsystem | Runtime | Logfile | Comment |
---|---|---|---|---|
+0 :ok: | reexec | 0m 48s | Docker mode activated. | |
_ Prechecks _ | ||||
+1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
+0 :ok: | codespell | 0m 1s | codespell was not available. | |
+0 :ok: | detsecrets | 0m 1s | detect-secrets was not available. | |
+1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
-1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | |
_ trunk Compile Tests _ | ||||
+1 :green_heart: | mvninstall | 41m 13s | trunk passed | |
+1 :green_heart: | compile | 1m 42s | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
+1 :green_heart: | compile | 1m 31s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
+1 :green_heart: | checkstyle | 1m 24s | trunk passed | |
+1 :green_heart: | mvnsite | 1m 39s | trunk passed | |
+1 :green_heart: | javadoc | 1m 19s | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
+1 :green_heart: | javadoc | 1m 44s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
+1 :green_heart: | spotbugs | 3m 46s | trunk passed | |
+1 :green_heart: | shadedclient | 26m 27s | branch has no errors when building and testing our client artifacts. | |
_ Patch Compile Tests _ | ||||
+1 :green_heart: | mvninstall | 1m 24s | the patch passed | |
+1 :green_heart: | compile | 1m 30s | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
+1 :green_heart: | javac | 1m 30s | the patch passed | |
+1 :green_heart: | compile | 1m 19s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
+1 :green_heart: | javac | 1m 19s | the patch passed | |
+1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
+1 :green_heart: | checkstyle | 1m 6s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 777 unchanged - 39 fixed = 777 total (was 816) | |
+1 :green_heart: | mvnsite | 1m 26s | the patch passed | |
+1 :green_heart: | javadoc | 0m 58s | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
+1 :green_heart: | javadoc | 1m 30s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
+1 :green_heart: | spotbugs | 3m 33s | the patch passed | |
+1 :green_heart: | shadedclient | 26m 18s | patch has no errors when building and testing our client artifacts. | |
_ Other Tests _ | ||||
+1 :green_heart: | unit | 342m 5s | hadoop-hdfs in the patch passed. | |
+1 :green_heart: | asflicense | 0m 59s | The patch does not generate ASF License warnings. | |
460m 59s |
Subsystem | Report/Notes |
---|---|
Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4668/3/artifact/out/Dockerfile |
GITHUB PR | https://github.com/apache/hadoop/pull/4668 |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
uname | Linux f1e4004c0e15 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | dev-support/bin/hadoop.sh |
git revision | trunk / 5e1e30451f406e776e66135f7d00265c763345a0 |
Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4668/3/testReport/ |
Max. process+thread count | 1961 (vs. ulimit of 5500) |
modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4668/3/console |
versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
@goiri @slfan1989 Master, I have rebased this patch based on the latest trunk. Can you help me merge this PR into trunk?
:broken_heart: -1 overall
Vote | Subsystem | Runtime | Logfile | Comment |
---|---|---|---|---|
+0 :ok: | reexec | 1m 0s | Docker mode activated. | |
_ Prechecks _ | ||||
+1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | |
+0 :ok: | codespell | 0m 1s | codespell was not available. | |
+0 :ok: | detsecrets | 0m 1s | detect-secrets was not available. | |
+1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
-1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | |
_ trunk Compile Tests _ | ||||
+1 :green_heart: | mvninstall | 38m 11s | trunk passed | |
+1 :green_heart: | compile | 1m 45s | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
+1 :green_heart: | compile | 1m 35s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
+1 :green_heart: | checkstyle | 1m 29s | trunk passed | |
+1 :green_heart: | mvnsite | 1m 43s | trunk passed | |
+1 :green_heart: | javadoc | 1m 25s | trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
+1 :green_heart: | javadoc | 1m 47s | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
+1 :green_heart: | spotbugs | 3m 43s | trunk passed | |
+1 :green_heart: | shadedclient | 23m 10s | branch has no errors when building and testing our client artifacts. | |
_ Patch Compile Tests _ | ||||
+1 :green_heart: | mvninstall | 1m 22s | the patch passed | |
+1 :green_heart: | compile | 1m 25s | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
+1 :green_heart: | javac | 1m 25s | the patch passed | |
+1 :green_heart: | compile | 1m 19s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
+1 :green_heart: | javac | 1m 19s | the patch passed | |
+1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
+1 :green_heart: | checkstyle | 1m 6s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 777 unchanged - 39 fixed = 777 total (was 816) | |
+1 :green_heart: | mvnsite | 1m 23s | the patch passed | |
+1 :green_heart: | javadoc | 0m 57s | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
+1 :green_heart: | javadoc | 1m 30s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
+1 :green_heart: | spotbugs | 3m 17s | the patch passed | |
+1 :green_heart: | shadedclient | 22m 34s | patch has no errors when building and testing our client artifacts. | |
_ Other Tests _ | ||||
-1 :x: | unit | 432m 14s | /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | hadoop-hdfs in the patch passed. |
+1 :green_heart: | asflicense | 1m 14s | The patch does not generate ASF License warnings. | |
542m 48s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
Subsystem | Report/Notes |
---|---|
Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4668/4/artifact/out/Dockerfile |
GITHUB PR | https://github.com/apache/hadoop/pull/4668 |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
uname | Linux 1819cb96a7be 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | dev-support/bin/hadoop.sh |
git revision | trunk / ac8d9518b45235667b5e69bdf0f2c9cf7e635cbe |
Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4668/4/testReport/ |
Max. process+thread count | 2689 (vs. ulimit of 5500) |
modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4668/4/console |
versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.