hbase
hbase copied to clipboard
apache
HBASE-29682: Roll HMaster WAL in response to FlushMasterStoreRequest
I argue that in addition to flushing the master region, the flushMasterStore() RPC should also roll the WAL files being written by the HMaster.
In clusters with small numbers of DataNodes (less than 9, in my experience), a rolling restart of the DataNodes will break WAL writers, in both HMasters and RegionServers. This can be worked around on RegionServers by calling admin.rollWALWriter() on every RegionServer after restarting each DataNode. Currently, there is no equivalent way to do this to an HMaster. I think the most elegant solution is to augment admin.flushMasterStore().
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 12s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | |
| _ master Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 5m 10s | master passed | |
| +1 :green_heart: | compile | 4m 16s | master passed | |
| +1 :green_heart: | checkstyle | 1m 14s | master passed | |
| +1 :green_heart: | spotbugs | 2m 17s | master passed | |
| +1 :green_heart: | spotless | 1m 7s | branch has no errors when running spotless:check. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 4m 43s | the patch passed | |
| +1 :green_heart: | compile | 4m 16s | the patch passed | |
| +1 :green_heart: | javac | 4m 16s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 1m 10s | the patch passed | |
| +1 :green_heart: | spotbugs | 2m 22s | the patch passed | |
| +1 :green_heart: | hadoopcheck | 14m 25s | Patch does not cause any errors with Hadoop 3.3.6 3.4.1. | |
| +1 :green_heart: | spotless | 0m 58s | patch has no errors when running spotless:check. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | |
| 51m 13s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7407/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7407 |
| JIRA Issue | HBASE-29682 |
| Optional Tests | dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless |
| uname | Linux 375e5ee70007 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / b4c4ae9e535e3b8b6e5a80abd869ac03b4e10585 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 71 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7407/1/console |
| versions | git=2.34.1 maven=3.9.8 spotbugs=4.7.3 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
Which WAL implementation do you use? We should be able to deal with datanode restart. The data which has been hflushed should be OK, and once there is a write error, we will try rolling the WAL and then write the data out again.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 12s | Docker mode activated. | |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck | |
| _ Prechecks _ | ||||
| _ master Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 4m 44s | master passed | |
| +1 :green_heart: | compile | 1m 16s | master passed | |
| +1 :green_heart: | javadoc | 0m 36s | master passed | |
| +1 :green_heart: | shadedjars | 7m 0s | branch has no errors when building our shaded downstream artifacts. | |
| _ Patch Compile Tests _ | ||||
| +1 :green_heart: | mvninstall | 4m 19s | the patch passed | |
| +1 :green_heart: | compile | 1m 13s | the patch passed | |
| +1 :green_heart: | javac | 1m 13s | the patch passed | |
| +1 :green_heart: | javadoc | 0m 35s | the patch passed | |
| +1 :green_heart: | shadedjars | 6m 51s | patch has no errors when building our shaded downstream artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 454m 18s | hbase-server in the patch passed. | |
| 485m 11s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7407/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/7407 |
| JIRA Issue | HBASE-29682 |
| Optional Tests | javac javadoc unit compile shadedjars |
| uname | Linux 2ca77c563354 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / b4c4ae9e535e3b8b6e5a80abd869ac03b4e10585 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7407/1/testReport/ |
| Max. process+thread count | 3093 (vs. ulimit of 30000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7407/1/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
We're not setting hbase.wal.provider, so in theory we ought to be using AsyncFSWAL, but the below stacktrace suggests we're using FSHLog, which I can't explain.
The problem we see in HMasters is this:
2025-10-29T18:36:31,234 [RpcServer.priority.RWQ.Fifo.read.handler=18,queue=1,port=60000] WARN org.apache.hadoop.hbase.master.MasterRpcServices: na1-few-cyan-clam.iad03.hubinternal.net,60020,1761675985630 reported a fatal error:
***** ABORTING region server na1-few-cyan-clam.iad03.hubinternal.net,60020,1761675985630: Failed log close in log roller *****
Cause:
org.apache.hadoop.hbase.regionserver.wal.FailedLogCloseException: hdfs://joke-hb2-a-qa:8020/hbase/WALs/na1-few-cyan-clam.iad03.hubinternal.net,60020,1761675985630/na1-few-cyan-clam.iad03.hubinternal.net%2C60020%2C1761675985630.1761761228611, unflushedEntries=7
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.doReplaceWriter(FSHLog.java:428)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.doReplaceWriter(FSHLog.java:69)
at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.lambda$replaceWriter$6(AbstractFSWAL.java:867)
at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)
at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.replaceWriter(AbstractFSWAL.java:866)
at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriterInternal(AbstractFSWAL.java:922)
at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.lambda$rollWriter$8(AbstractFSWAL.java:952)
at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)
at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:952)
at org.apache.hadoop.hbase.wal.AbstractWALRoller$RollController.rollWal(AbstractWALRoller.java:305)
at org.apache.hadoop.hbase.wal.AbstractWALRoller.run(AbstractWALRoller.java:211)
Caused by: org.apache.hadoop.hbase.regionserver.wal.FailedSyncBeforeLogCloseException: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=215213199, requesting roll of WAL
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.checkIfSyncFailed(FSHLog.java:885)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.waitSafePoint(FSHLog.java:901)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.doReplaceWriter(FSHLog.java:365)
... 10 more
Caused by: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=215213199, requesting roll of WAL
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1183)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1056)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:959)
at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[172.22.241.95:50010,DS-4007a5dc-1679-46c1-8e0f-31178d1c5c65,DISK], DatanodeInfoWithStorage[172.22.66.247:50010,DS-e060a81f-3473-4625-b81a-776dc93622c3,DISK]], original=[DatanodeInfoWithStorage[172.22.241.95:50010,DS-4007a5dc-1679-46c1-8e0f-31178d1c5c65,DISK], DatanodeInfoWithStorage[172.22.66.247:50010,DS-e060a81f-3473-4625-b81a-776dc93622c3,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.
at org.apache.hadoop.hdfs.DataStreamer.findNewDatanode(DataStreamer.java:1358)
at org.apache.hadoop.hdfs.DataStreamer.addDatanode2ExistingPipeline(DataStreamer.java:1426)
at org.apache.hadoop.hdfs.DataStreamer.handleDatanodeReplacement(DataStreamer.java:1652)
at org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1553)
at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1535)
at org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1311)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:671)
Since several DataNodes have recently restarted, they get added to the DFSClient's bad node list, and once enough DataNodes are on the bad nodes list, there are none left to write to. This makes the HMaster restart.
We're not setting
hbase.wal.provider, so in theory we ought to be usingAsyncFSWAL, but the below stacktrace suggests we're usingFSHLog, which I can't explain.The problem we see in HMasters is this:
2025-10-29T18:36:31,234 [RpcServer.priority.RWQ.Fifo.read.handler=18,queue=1,port=60000] WARN org.apache.hadoop.hbase.master.MasterRpcServices: na1-few-cyan-clam.iad03.hubinternal.net,60020,1761675985630 reported a fatal error: ***** ABORTING region server na1-few-cyan-clam.iad03.hubinternal.net,60020,1761675985630: Failed log close in log roller ***** Cause: org.apache.hadoop.hbase.regionserver.wal.FailedLogCloseException: hdfs://joke-hb2-a-qa:8020/hbase/WALs/na1-few-cyan-clam.iad03.hubinternal.net,60020,1761675985630/na1-few-cyan-clam.iad03.hubinternal.net%2C60020%2C1761675985630.1761761228611, unflushedEntries=7 at org.apache.hadoop.hbase.regionserver.wal.FSHLog.doReplaceWriter(FSHLog.java:428) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.doReplaceWriter(FSHLog.java:69) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.lambda$replaceWriter$6(AbstractFSWAL.java:867) at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.replaceWriter(AbstractFSWAL.java:866) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriterInternal(AbstractFSWAL.java:922) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.lambda$rollWriter$8(AbstractFSWAL.java:952) at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:952) at org.apache.hadoop.hbase.wal.AbstractWALRoller$RollController.rollWal(AbstractWALRoller.java:305) at org.apache.hadoop.hbase.wal.AbstractWALRoller.run(AbstractWALRoller.java:211) Caused by: org.apache.hadoop.hbase.regionserver.wal.FailedSyncBeforeLogCloseException: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=215213199, requesting roll of WAL at org.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.checkIfSyncFailed(FSHLog.java:885) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.waitSafePoint(FSHLog.java:901) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.doReplaceWriter(FSHLog.java:365) ... 10 more Caused by: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=215213199, requesting roll of WAL at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1183) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1056) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:959) at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) at java.base/java.lang.Thread.run(Thread.java:1583) Caused by: java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[172.22.241.95:50010,DS-4007a5dc-1679-46c1-8e0f-31178d1c5c65,DISK], DatanodeInfoWithStorage[172.22.66.247:50010,DS-e060a81f-3473-4625-b81a-776dc93622c3,DISK]], original=[DatanodeInfoWithStorage[172.22.241.95:50010,DS-4007a5dc-1679-46c1-8e0f-31178d1c5c65,DISK], DatanodeInfoWithStorage[172.22.66.247:50010,DS-e060a81f-3473-4625-b81a-776dc93622c3,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DataStreamer.findNewDatanode(DataStreamer.java:1358) at org.apache.hadoop.hdfs.DataStreamer.addDatanode2ExistingPipeline(DataStreamer.java:1426) at org.apache.hadoop.hdfs.DataStreamer.handleDatanodeReplacement(DataStreamer.java:1652) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1553) at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1535) at org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1311) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:671)Since several DataNodes have recently restarted, they get added to the DFSClient's bad node list, and once enough DataNodes are on the bad nodes list, there are none left to write to. This makes the HMaster restart.
Seems we have already started a log roll here? Then I do not think issuing a log roll manually can fix the problem?