alluxio icon indicating copy to clipboard operation
alluxio copied to clipboard

Published 20 hours ago • verified publisher icon Alluxio

master stopped when run max-file-stress-bench to full the disk of 'alluxio.master.metastore.dir'

Open wwq2333 opened this issue 2 years ago • 0 comments

Alluxio Version: 2.8.0

Describe the bug when run alluxio.stress.cli.MaxFileBench, if the disk of metastore.dir is full, the master will stop (logs lost leadership)

To Reproduce

# prepare metastore.dir (limited 50M, reproducing problems faster)
mount -t tmpfs -o size=50M tmpfs memdisk

# Change the configuration
alluxio.master.metastore=ROCKS
alluxio.master.metastore.dir=/data/memdisk
alluxio.master.metastore.inode.cache.max.size=0

# start all
./bin/alluxio-start.sh all

# run MaxFileBench
./bin/alluxio runClass alluxio.stress.cli.MaxFileBench --base alluxio://*.*.*.*:19998/stress-maxfile --write-type THROUGH

# tail log to observe the failure, and use ps to confirm master process is stopped
tail -f logs/master.log

Expected behavior

master can continue to serve for the requests whose target file meta is stored, rather than stop directly.

Urgency a little

Are you planning to fix it No

Additional context master log as follow

2022-08-08 18:39:41,676 INFO  NettyUtils - EPOLL_MODE is available

The following is running alluxio.stress.cli.MaxFileBench
<====

2022-08-08 18:41:12,739 WARN  InodeSyncStream - Failed to sync metadata on root path InodeSyncStream{rootPath=LockingScheme{path=/stress-maxfile/files, desiredLockPattern=READ, shouldSync=false}, descendantType=NONE, commonOptions=syncIntervalMs: -1
ttl: -1
ttlAction: DELETE
, forceSync=true, isGetFileInfo=true}: alluxio.exception.FileDoesNotExistException: Path "/stress-maxfile/files" does not exist.
2022-08-08 18:41:17,885 INFO  SegmentedRaftLogWorker - *.*.*.*_19200@group-ABB3109A44C1-SegmentedRaftLogWorker: Rolling segment log-0_992 to index:992
2022-08-08 18:41:17,888 INFO  SegmentedRaftLogWorker - *.*.*.*_19200@group-ABB3109A44C1-SegmentedRaftLogWorker: Rolled log segment from /data/wwq/alluxio-2.8.0/journal/raft/02511d47-d67c-49a3-9011-abb3109a44c1/current/log_inprogress_0 to /data/wwq/alluxio-2.8.0/journal/raft/02511d47-d67c-49a3-9011-abb3109a44c1/current/log_0-992
2022-08-08 18:41:17,894 INFO  SegmentedRaftLogWorker - *.*.*.*_19200@group-ABB3109A44C1-SegmentedRaftLogWorker: created new log segment /data/wwq/alluxio-2.8.0/journal/raft/02511d47-d67c-49a3-9011-abb3109a44c1/current/log_inprogress_993
2022-08-08 18:41:19,061 INFO  SegmentedRaftLogWorker - *.*.*.*_19200@group-ABB3109A44C1-SegmentedRaftLogWorker: Rolling segment log-993_1988 to index:1988
2022-08-08 18:41:19,062 INFO  SegmentedRaftLogWorker - *.*.*.*_19200@group-ABB3109A44C1-SegmentedRaftLogWorker: Rolled log segment from /data/wwq/alluxio-2.8.0/journal/raft/02511d47-d67c-49a3-9011-abb3109a44c1/current/log_inprogress_993 to /data/wwq/alluxio-2.8.0/journal/raft/02511d47-d67c-49a3-9011-abb3109a44c1/current/log_993-1988
2022-08-08 18:41:19,066 INFO  SegmentedRaftLogWorker - *.*.*.*_19200@group-ABB3109A44C1-SegmentedRaftLogWorker: created new log segment /data/wwq/alluxio-2.8.0/journal/raft/02511d47-d67c-49a3-9011-abb3109a44c1/current/log_inprogress_1989
2022-08-08 18:41:20,202 INFO  SegmentedRaftLogWorker - *.*.*.*_19200@group-ABB3109A44C1-SegmentedRaftLogWorker: Rolling segment log-1989_2962 to index:2962
2022-08-08 18:41:20,203 INFO  SegmentedRaftLogWorker - *.*.*.*_19200@group-ABB3109A44C1-SegmentedRaftLogWorker: Rolled log segment from /data/wwq/alluxio-2.8.0/journal/raft/02511d47-d67c-49a3-9011-abb3109a44c1/current/log_inprogress_1989 to /data/wwq/alluxio-2.8.0/journal/raft/02511d47-d67c-49a3-9011-abb3109a44c1/current/log_1989-2962
2022-08-08 18:41:20,207 INFO  SegmentedRaftLogWorker - *.*.*.*_19200@group-ABB3109A44C1-SegmentedRaftLogWorker: created new log segment /data/wwq/alluxio-2.8.0/journal/raft/02511d47-d67c-49a3-9011-abb3109a44c1/current/log_inprogress_2963
2022-08-08 18:41:21,317 INFO  SegmentedRaftLogWorker - *.*.*.*_19200@group-ABB3109A44C1-SegmentedRaftLogWorker: Rolling segment log-2963_3842 to index:3842
2022-08-08 18:41:21,317 INFO  SegmentedRaftLogWorker - *.*.*.*_19200@group-ABB3109A44C1-SegmentedRaftLogWorker: Rolled log segment from /data/wwq/alluxio-2.8.0/journal/raft/02511d47-d67c-49a3-9011-abb3109a44c1/current/log_inprogress_2963 to /data/wwq/alluxio-2.8.0/journal/raft/02511d47-d67c-49a3-9011-abb3109a44c1/current/log_2963-3842
2022-08-08 18:41:21,322 INFO  SegmentedRaftLogWorker - *.*.*.*_19200@group-ABB3109A44C1-SegmentedRaftLogWorker: created new log segment /data/wwq/alluxio-2.8.0/journal/raft/02511d47-d67c-49a3-9011-abb3109a44c1/current/log_inprogress_3843
2022-08-08 18:41:22,542 INFO  SegmentedRaftLogWorker - *.*.*.*_19200@group-ABB3109A44C1-SegmentedRaftLogWorker: Rolling segment log-3843_4732 to index:4732
2022-08-08 18:41:22,543 INFO  SegmentedRaftLogWorker - *.*.*.*_19200@group-ABB3109A44C1-SegmentedRaftLogWorker: Rolled log segment from /data/wwq/alluxio-2.8.0/journal/raft/02511d47-d67c-49a3-9011-abb3109a44c1/current/log_inprogress_3843 to /data/wwq/alluxio-2.8.0/journal/raft/02511d47-d67c-49a3-9011-abb3109a44c1/current/log_3843-4732
2022-08-08 18:41:22,548 INFO  SegmentedRaftLogWorker - *.*.*.*_19200@group-ABB3109A44C1-SegmentedRaftLogWorker: created new log segment /data/wwq/alluxio-2.8.0/journal/raft/02511d47-d67c-49a3-9011-abb3109a44c1/current/log_inprogress_4733
2022-08-08 18:41:23,968 INFO  SegmentedRaftLogWorker - *.*.*.*_19200@group-ABB3109A44C1-SegmentedRaftLogWorker: Rolling segment log-4733_5764 to index:5764
2022-08-08 18:41:23,968 INFO  SegmentedRaftLogWorker - *.*.*.*_19200@group-ABB3109A44C1-SegmentedRaftLogWorker: Rolled log segment from /data/wwq/alluxio-2.8.0/journal/raft/02511d47-d67c-49a3-9011-abb3109a44c1/current/log_inprogress_4733 to /data/wwq/alluxio-2.8.0/journal/raft/02511d47-d67c-49a3-9011-abb3109a44c1/current/log_4733-5764
2022-08-08 18:41:23,973 INFO  SegmentedRaftLogWorker - *.*.*.*_19200@group-ABB3109A44C1-SegmentedRaftLogWorker: created new log segment /data/wwq/alluxio-2.8.0/journal/raft/02511d47-d67c-49a3-9011-abb3109a44c1/current/log_inprogress_5765
2022-08-08 18:41:25,565 INFO  SegmentedRaftLogWorker - *.*.*.*_19200@group-ABB3109A44C1-SegmentedRaftLogWorker: Rolling segment log-5765_6678 to index:6678
2022-08-08 18:41:25,566 INFO  SegmentedRaftLogWorker - *.*.*.*_19200@group-ABB3109A44C1-SegmentedRaftLogWorker: Rolled log segment from /data/wwq/alluxio-2.8.0/journal/raft/02511d47-d67c-49a3-9011-abb3109a44c1/current/log_inprogress_5765 to /data/wwq/alluxio-2.8.0/journal/raft/02511d47-d67c-49a3-9011-abb3109a44c1/current/log_5765-6678
2022-08-08 18:41:25,571 INFO  SegmentedRaftLogWorker - *.*.*.*_19200@group-ABB3109A44C1-SegmentedRaftLogWorker: created new log segment /data/wwq/alluxio-2.8.0/journal/raft/02511d47-d67c-49a3-9011-abb3109a44c1/current/log_inprogress_6679
2022-08-08 18:41:26,715 INFO  SegmentedRaftLogWorker - *.*.*.*_19200@group-ABB3109A44C1-SegmentedRaftLogWorker: Rolling segment log-6679_7606 to index:7606
2022-08-08 18:41:26,717 INFO  SegmentedRaftLogWorker - *.*.*.*_19200@group-ABB3109A44C1-SegmentedRaftLogWorker: Rolled log segment from /data/wwq/alluxio-2.8.0/journal/raft/02511d47-d67c-49a3-9011-abb3109a44c1/current/log_inprogress_6679 to /data/wwq/alluxio-2.8.0/journal/raft/02511d47-d67c-49a3-9011-abb3109a44c1/current/log_6679-7606
2022-08-08 18:41:26,722 INFO  SegmentedRaftLogWorker - *.*.*.*_19200@group-ABB3109A44C1-SegmentedRaftLogWorker: created new log segment /data/wwq/alluxio-2.8.0/journal/raft/02511d47-d67c-49a3-9011-abb3109a44c1/current/log_inprogress_7607
2022-08-08 18:41:27,623 ERROR InodeTreePersistentState - Fatal error: Failed to apply MutableInodeFile{id=5615686516735, name=1293, parentId=96, creationTimeMs=1659955287567, pinned=false, deleted=false, ttl=-1, ttlAction=DELETE, directory=false, persistenceState=NOT_PERSISTED, lastModificationTimeMs=1659955287566, lastAccessTimeMs=1659955287566, owner=SDE, group=toc, permission=420, ufsFingerprint=, mediatypes=[], xAttr={}, blocks=[], blockContainerId=334720, blockSizeBytes=67108864, cacheable=true, completed=false, persistJobId=-1, persistenceWaitTime=1659955287567, replicationDurable=1, replicationMax=-1, replicationMin=0, tempUfsPath=, length=0}
java.lang.RuntimeException: org.rocksdb.RocksDBException: While appending to file: /data/memdisk/inodes/000014.sst: No space left on device
	at alluxio.master.metastore.rocks.RocksInodeStore.writeInode(RocksInodeStore.java:116)
	at alluxio.master.metastore.InodeStore.writeNewInode(InodeStore.java:119)
	at alluxio.master.file.meta.InodeTreePersistentState.applyCreateInode(InodeTreePersistentState.java:644)
	at alluxio.master.file.meta.InodeTreePersistentState.applyAndJournal(InodeTreePersistentState.java:355)
	at alluxio.master.file.meta.InodeTree.createPath(InodeTree.java:934)
	at alluxio.master.file.DefaultFileSystemMaster.createFileInternal(DefaultFileSystemMaster.java:1732)
	at alluxio.master.file.DefaultFileSystemMaster.createFile(DefaultFileSystemMaster.java:1710)
	at alluxio.master.file.FileSystemMasterClientServiceHandler.lambda$createFile$5(FileSystemMasterClientServiceHandler.java:194)
	at alluxio.RpcUtils.callAndReturn(RpcUtils.java:122)
	at alluxio.RpcUtils.call(RpcUtils.java:83)
	at alluxio.RpcUtils.call(RpcUtils.java:58)
	at alluxio.master.file.FileSystemMasterClientServiceHandler.createFile(FileSystemMasterClientServiceHandler.java:191)
	at alluxio.grpc.FileSystemMasterClientServiceGrpc$MethodHandlers.invoke(FileSystemMasterClientServiceGrpc.java:2376)
	at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182)
	at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
	at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
	at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
	at alluxio.security.authentication.ClientIpAddressInjector$1.onHalfClose(ClientIpAddressInjector.java:57)
	at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
	at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
	at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
	at alluxio.security.authentication.AuthenticatedUserInjector$1.onHalfClose(AuthenticatedUserInjector.java:67)
	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:331)
	at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:797)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.rocksdb.RocksDBException: While appending to file: /data/memdisk/inodes/000014.sst: No space left on device
	at org.rocksdb.RocksDB.put(Native Method)
	at org.rocksdb.RocksDB.put(RocksDB.java:955)
	at alluxio.master.metastore.rocks.RocksInodeStore.writeInode(RocksInodeStore.java:113)
	... 28 more

2022-08-08 18:41:27,623 ERROR InodeTreePersistentState - Fatal error: Failed to apply MutableInodeFile{id=5615703293951, name=1327, parentId=147, creationTimeMs=1659955287567, pinned=false, deleted=false, ttl=-1, ttlAction=DELETE, directory=false, persistenceState=NOT_PERSISTED, lastModificationTimeMs=1659955287566, lastAccessTimeMs=1659955287566, owner=SDE, group=toc, permission=420, ufsFingerprint=, mediatypes=[], xAttr={}, blocks=[], blockContainerId=334721, blockSizeBytes=67108864, cacheable=true, completed=false, persistJobId=-1, persistenceWaitTime=1659955287567, replicationDurable=1, replicationMax=-1, replicationMin=0, tempUfsPath=, length=0}
java.lang.RuntimeException: org.rocksdb.RocksDBException: While appending to file: /data/memdisk/inodes/000014.sst: No space left on device
	at alluxio.master.metastore.rocks.RocksInodeStore.writeInode(RocksInodeStore.java:116)
	at alluxio.master.metastore.InodeStore.writeNewInode(InodeStore.java:119)
	at alluxio.master.file.meta.InodeTreePersistentState.applyCreateInode(InodeTreePersistentState.java:644)
	at alluxio.master.file.meta.InodeTreePersistentState.applyAndJournal(InodeTreePersistentState.java:355)
	at alluxio.master.file.meta.InodeTree.createPath(InodeTree.java:934)
	at alluxio.master.file.DefaultFileSystemMaster.createFileInternal(DefaultFileSystemMaster.java:1732)
	at alluxio.master.file.DefaultFileSystemMaster.createFile(DefaultFileSystemMaster.java:1710)
	at alluxio.master.file.FileSystemMasterClientServiceHandler.lambda$createFile$5(FileSystemMasterClientServiceHandler.java:194)
	at alluxio.RpcUtils.callAndReturn(RpcUtils.java:122)
	at alluxio.RpcUtils.call(RpcUtils.java:83)
	at alluxio.RpcUtils.call(RpcUtils.java:58)
	at alluxio.master.file.FileSystemMasterClientServiceHandler.createFile(FileSystemMasterClientServiceHandler.java:191)
	at alluxio.grpc.FileSystemMasterClientServiceGrpc$MethodHandlers.invoke(FileSystemMasterClientServiceGrpc.java:2376)
	at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182)
	at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
	at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
	at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
	at alluxio.security.authentication.ClientIpAddressInjector$1.onHalfClose(ClientIpAddressInjector.java:57)
	at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
	at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
	at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
	at alluxio.security.authentication.AuthenticatedUserInjector$1.onHalfClose(AuthenticatedUserInjector.java:67)
	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:331)
	at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:797)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.rocksdb.RocksDBException: While appending to file: /data/memdisk/inodes/000014.sst: No space left on device
	at org.rocksdb.RocksDB.put(Native Method)
	at org.rocksdb.RocksDB.put(RocksDB.java:955)
	at alluxio.master.metastore.rocks.RocksInodeStore.writeInode(RocksInodeStore.java:113)
	... 28 more

2022-08-08 18:41:27,623 ERROR InodeTreePersistentState - Fatal error: Failed to apply MutableInodeFile{id=5615669739519, name=1311, parentId=387, creationTimeMs=1659955287567, pinned=false, deleted=false, ttl=-1, ttlAction=DELETE, directory=false, persistenceState=NOT_PERSISTED, lastModificationTimeMs=1659955287566, lastAccessTimeMs=1659955287566, owner=SDE, group=toc, permission=420, ufsFingerprint=, mediatypes=[], xAttr={}, blocks=[], blockContainerId=334719, blockSizeBytes=67108864, cacheable=true, completed=false, persistJobId=-1, persistenceWaitTime=1659955287567, replicationDurable=1, replicationMax=-1, replicationMin=0, tempUfsPath=, length=0}
java.lang.RuntimeException: org.rocksdb.RocksDBException: While appending to file: /data/memdisk/inodes/000014.sst: No space left on device
	at alluxio.master.metastore.rocks.RocksInodeStore.writeInode(RocksInodeStore.java:116)
	at alluxio.master.metastore.InodeStore.writeNewInode(InodeStore.java:119)
	at alluxio.master.file.meta.InodeTreePersistentState.applyCreateInode(InodeTreePersistentState.java:644)
	at alluxio.master.file.meta.InodeTreePersistentState.applyAndJournal(InodeTreePersistentState.java:355)
	at alluxio.master.file.meta.InodeTree.createPath(InodeTree.java:934)
	at alluxio.master.file.DefaultFileSystemMaster.createFileInternal(DefaultFileSystemMaster.java:1732)
	at alluxio.master.file.DefaultFileSystemMaster.createFile(DefaultFileSystemMaster.java:1710)
	at alluxio.master.file.FileSystemMasterClientServiceHandler.lambda$createFile$5(FileSystemMasterClientServiceHandler.java:194)
	at alluxio.RpcUtils.callAndReturn(RpcUtils.java:122)
	at alluxio.RpcUtils.call(RpcUtils.java:83)
	at alluxio.RpcUtils.call(RpcUtils.java:58)
	at alluxio.master.file.FileSystemMasterClientServiceHandler.createFile(FileSystemMasterClientServiceHandler.java:191)
	at alluxio.grpc.FileSystemMasterClientServiceGrpc$MethodHandlers.invoke(FileSystemMasterClientServiceGrpc.java:2376)
	at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182)
	at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
	at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
	at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
	at alluxio.security.authentication.ClientIpAddressInjector$1.onHalfClose(ClientIpAddressInjector.java:57)
	at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
	at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
	at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
	at alluxio.security.authentication.AuthenticatedUserInjector$1.onHalfClose(AuthenticatedUserInjector.java:67)
	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:331)
	at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:797)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.rocksdb.RocksDBException: While appending to file: /data/memdisk/inodes/000014.sst: No space left on device
	at org.rocksdb.RocksDB.put(Native Method)
	at org.rocksdb.RocksDB.put(RocksDB.java:955)
	at alluxio.master.metastore.rocks.RocksInodeStore.writeInode(RocksInodeStore.java:113)
	... 28 more

2022-08-08 18:41:27,626 INFO  FaultTolerantAlluxioMasterProcess - Stopping...
2022-08-08 18:41:27,628 INFO  AlluxioMasterProcess - Stopping...
2022-08-08 18:41:27,629 ERROR InodeTreePersistentState - Fatal error: Failed to apply id: 303
last_modification_time_ms: 1659955287568
last_access_time_ms: 1659955287568

================================
....... Lots of logs like above
================================

2022-08-08 18:41:27,774 ERROR InodeTreePersistentState - Fatal error: Failed to apply id: 254
last_modification_time_ms: 1659955287567
last_access_time_ms: 1659955287567

java.lang.RuntimeException: org.rocksdb.RocksDBException: While appending to file: /data/memdisk/inodes/000014.sst: No space left on device
	at alluxio.master.metastore.rocks.RocksInodeStore.writeInode(RocksInodeStore.java:116)
	at alluxio.master.file.meta.InodeTreePersistentState.applyUpdateInode(InodeTreePersistentState.java:487)
	at alluxio.master.file.meta.InodeTreePersistentState.applyAndJournal(InodeTreePersistentState.java:300)
	at alluxio.master.file.meta.InodeTree.createPath(InodeTree.java:786)
	at alluxio.master.file.DefaultFileSystemMaster.createFileInternal(DefaultFileSystemMaster.java:1732)
	at alluxio.master.file.DefaultFileSystemMaster.createFile(DefaultFileSystemMaster.java:1710)
	at alluxio.master.file.FileSystemMasterClientServiceHandler.lambda$createFile$5(FileSystemMasterClientServiceHandler.java:194)
	at alluxio.RpcUtils.callAndReturn(RpcUtils.java:122)
	at alluxio.RpcUtils.call(RpcUtils.java:83)
	at alluxio.RpcUtils.call(RpcUtils.java:58)
	at alluxio.master.file.FileSystemMasterClientServiceHandler.createFile(FileSystemMasterClientServiceHandler.java:191)
	at alluxio.grpc.FileSystemMasterClientServiceGrpc$MethodHandlers.invoke(FileSystemMasterClientServiceGrpc.java:2376)
	at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182)
	at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
	at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
	at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
	at alluxio.security.authentication.ClientIpAddressInjector$1.onHalfClose(ClientIpAddressInjector.java:57)
	at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
	at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
	at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
	at alluxio.security.authentication.AuthenticatedUserInjector$1.onHalfClose(AuthenticatedUserInjector.java:67)
	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:331)
	at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:797)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.rocksdb.RocksDBException: While appending to file: /data/memdisk/inodes/000014.sst: No space left on device
	at org.rocksdb.RocksDB.put(Native Method)
	at org.rocksdb.RocksDB.put(RocksDB.java:955)
	at alluxio.master.metastore.rocks.RocksInodeStore.writeInode(RocksInodeStore.java:113)
	... 27 more

2022-08-08 18:41:27,774 ERROR InodeTreePersistentState - Fatal error: Failed to apply id: 353
last_modification_time_ms: 1659955287573
last_access_time_ms: 1659955287573

java.lang.RuntimeException: org.rocksdb.RocksDBException: While appending to file: /data/memdisk/inodes/000014.sst: No space left on device
	at alluxio.master.metastore.rocks.RocksInodeStore.writeInode(RocksInodeStore.java:116)
	at alluxio.master.file.meta.InodeTreePersistentState.applyUpdateInode(InodeTreePersistentState.java:487)
	at alluxio.master.file.meta.InodeTreePersistentState.applyAndJournal(InodeTreePersistentState.java:300)
	at alluxio.master.file.meta.InodeTree.createPath(InodeTree.java:786)
	at alluxio.master.file.DefaultFileSystemMaster.createFileInternal(DefaultFileSystemMaster.java:1732)
	at alluxio.master.file.DefaultFileSystemMaster.createFile(DefaultFileSystemMaster.java:1710)
	at alluxio.master.file.FileSystemMasterClientServiceHandler.lambda$createFile$5(FileSystemMasterClientServiceHandler.java:194)
	at alluxio.RpcUtils.callAndReturn(RpcUtils.java:122)
	at alluxio.RpcUtils.call(RpcUtils.java:83)
	at alluxio.RpcUtils.call(RpcUtils.java:58)
	at alluxio.master.file.FileSystemMasterClientServiceHandler.createFile(FileSystemMasterClientServiceHandler.java:191)
	at alluxio.grpc.FileSystemMasterClientServiceGrpc$MethodHandlers.invoke(FileSystemMasterClientServiceGrpc.java:2376)
	at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182)
	at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
	at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
	at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
	at alluxio.security.authentication.ClientIpAddressInjector$1.onHalfClose(ClientIpAddressInjector.java:57)
	at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
	at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
	at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
	at alluxio.security.authentication.AuthenticatedUserInjector$1.onHalfClose(AuthenticatedUserInjector.java:67)
	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:331)
	at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:797)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.rocksdb.RocksDBException: While appending to file: /data/memdisk/inodes/000014.sst: No space left on device
	at org.rocksdb.RocksDB.put(Native Method)
	at org.rocksdb.RocksDB.put(RocksDB.java:955)
	at alluxio.master.metastore.rocks.RocksInodeStore.writeInode(RocksInodeStore.java:113)
	... 27 more

2022-08-08 18:41:27,797 INFO  AlluxioMasterProcess - Alluxio master ended  (lost leadership)
2022-08-08 18:42:27,808 INFO  AbstractConnector - Stopped ServerConnector@1a4347ef{HTTP/1.1, (http/1.1)}{0.0.0.0:19999}
2022-08-08 18:42:27,857 INFO  ContextHandler - Stopped o.e.j.s.ServletContextHandler@74c78f98{/,file:///data/wwq/alluxio-2.8.0/webui/master/build/,STOPPED}
2022-08-08 18:42:27,858 INFO  ContextHandler - Stopped o.e.j.s.ServletContextHandler@545efb7d{/metrics/prometheus,null,STOPPED}
2022-08-08 18:42:27,859 INFO  ContextHandler - Stopped o.e.j.s.ServletContextHandler@61593e9b{/metrics/json,null,STOPPED}
2022-08-08 18:42:27,861 WARN  JvmPauseMonitor - JVM pause monitor interrupted during sleep.
2022-08-08 18:42:27,861 INFO  RaftJournalSystem - Shutting down raft journal
2022-08-08 18:42:27,862 INFO  RaftJournalWriter - Closing journal writer. Last sequence numbers written/submitted/committed: 670607/670607/670607
2022-08-08 18:42:27,863 INFO  RaftServer - *.*.*.*_19200: close
2022-08-08 18:42:27,866 INFO  JmxRegister - Successfully un-registered JMX Bean with object name Ratis:service=RaftServer,group=group-ABB3109A44C1,id=*.*.*.*_19200
2022-08-08 18:42:27,866 INFO  RoleInfo - *.*.*.*_19200: shutdown *.*.*.*_19200@group-ABB3109A44C1-LeaderStateImpl
2022-08-08 18:42:27,867 INFO  PendingRequests - *.*.*.*_19200@group-ABB3109A44C1-PendingRequests: sendNotLeaderResponses
2022-08-08 18:42:27,867 INFO  AbstractPrimarySelector - Primary selector transitioning to STANDBY
2022-08-08 18:42:27,873 INFO  ProcessUtils - Stopping Alluxio master @*.*.*.*:19998.
2022-08-08 18:42:27,876 INFO  RatisMetrics - Unregistering Metrics Registry : ratis.log_appender.*.*.*.*_19200@group-ABB3109A44C1
2022-08-08 18:42:27,877 INFO  RatisMetrics - Unregistering Metrics Registry : ratis.server.*.*.*.*_19200@group-ABB3109A44C1
2022-08-08 18:42:27,877 INFO  StateMachineUpdater - *.*.*.*_19200@group-ABB3109A44C1-StateMachineUpdater: set stopIndex = 8306
2022-08-08 18:42:27,878 INFO  JournalStateMachine - Skip taking snapshot because raft server is not in running state: current state is CLOSING.
2022-08-08 18:42:27,878 INFO  RatisMetrics - Unregistering Metrics Registry : ratis.state_machine.*.*.*.*_19200@group-ABB3109A44C1
2022-08-08 18:42:27,879 INFO  SegmentedRaftLogWorker - *.*.*.*_19200@group-ABB3109A44C1-SegmentedRaftLogWorker was interrupted, exiting. There are 0 tasks remaining in the queue.
2022-08-08 18:42:27,881 INFO  SegmentedRaftLogWorker - *.*.*.*_19200@group-ABB3109A44C1-SegmentedRaftLogWorker close()
2022-08-08 18:42:27,882 INFO  RatisMetrics - Unregistering Metrics Registry : ratis.log_worker.*.*.*.*_19200@group-ABB3109A44C1
2022-08-08 18:42:27,883 INFO  RatisMetrics - Unregistering Metrics Registry : ratis.leader_election.*.*.*.*_19200@group-ABB3109A44C1
2022-08-08 18:42:27,883 INFO  RatisMetrics - Unregistering Metrics Registry : ratis.server.*.*.*.*_19200@group-ABB3109A44C1
2022-08-08 18:42:27,883 INFO  GrpcService - *.*.*.*_19200: shutdown server with port 19200 now
2022-08-08 18:42:27,910 INFO  GrpcService - *.*.*.*_19200: shutdown server with port 19200 successfully
2022-08-08 18:42:27,911 INFO  JvmPauseMonitor - JvmPauseMonitor-*.*.*.*_19200: Stopped
2022-08-08 18:42:27,911 INFO  RaftJournalSystem - Journal shutdown complete
2022-08-08 18:42:27,911 INFO  AlluxioMasterProcess - Closing all masters.
2022-08-08 18:42:27,911 INFO  AbstractMaster - TableMaster: Stopping primary master.
2022-08-08 18:42:27,912 INFO  AbstractMaster - TableMaster: Stopped primary master.
2022-08-08 18:42:27,912 INFO  AbstractMaster - MetaMaster: Stopping primary master.
2022-08-08 18:42:27,912 INFO  AbstractMaster - MetaMaster: Stopped primary master.
2022-08-08 18:42:27,912 INFO  AbstractMaster - FileSystemMaster: Stopping primary master.
2022-08-08 18:42:27,912 INFO  AbstractMaster - FileSystemMaster: Stopped primary master.
2022-08-08 18:42:27,912 INFO  RocksInodeStore - Closing RocksInodeStore and recycling all RocksDB JNI objects
2022-08-08 18:42:27,913 INFO  RocksStore - Closing InodeStore rocks database
2022-08-08 18:42:27,914 INFO  RocksStore - Closed store at /data/memdisk/inodes
2022-08-08 18:42:27,914 INFO  RocksInodeStore - RocksInodeStore closed
2022-08-08 18:42:27,915 INFO  AbstractMaster - BlockMaster: Stopping primary master.
2022-08-08 18:42:27,915 INFO  AbstractMaster - BlockMaster: Stopped primary master.
2022-08-08 18:42:27,915 INFO  RocksBlockStore - Closing RocksBlockStore and recycling all RocksDB JNI objects
2022-08-08 18:42:27,915 INFO  RocksStore - Closing BlockStore rocks database
2022-08-08 18:42:27,916 INFO  RocksStore - Closed store at /data/memdisk/blocks
2022-08-08 18:42:27,916 INFO  RocksBlockStore - RocksBlockStore closed
2022-08-08 18:42:27,916 INFO  AbstractMaster - MetricsMaster: Stopping primary master.
2022-08-08 18:42:27,916 INFO  AbstractMaster - MetricsMaster: Stopped primary master.
2022-08-08 18:42:27,916 INFO  AlluxioMasterProcess - Closed all masters.
2022-08-08 18:42:27,916 INFO  AlluxioMasterProcess - Stopped.

wwq2333 avatar Aug 08 '22 12:08 wwq2333