Paper
Paper copied to clipboard
PaperMC
Server Crash
Stack trace
[18:50:19 ERROR]: Encountered an unexpected exception
net.minecraft.ReportedException: Exception ticking world
at net.minecraft.server.MinecraftServer.b(MinecraftServer.java:1568) ~[patched_1.17.1.jar:git-Paper-186]
at net.minecraft.server.dedicated.DedicatedServer.b(DedicatedServer.java:479) ~[patched_1.17.1.jar:git-Paper-186]
at net.minecraft.server.MinecraftServer.a(MinecraftServer.java:1406) ~[patched_1.17.1.jar:git-Paper-186]
at net.minecraft.server.MinecraftServer.x(MinecraftServer.java:1188) ~[patched_1.17.1.jar:git-Paper-186]
at net.minecraft.server.MinecraftServer.lambda$spin$0(MinecraftServer.java:316) ~[patched_1.17.1.jar:git-Paper-186]
at java.lang.Thread.run(Thread.java:831) [?:?]
Caused by: java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1013) ~[?:?]
at java.util.ArrayList$Itr.next(ArrayList.java:967) ~[?:?]
at com.google.common.collect.Iterators$3.next(Iterators.java:174) ~[patched_1.17.1.jar:git-Paper-186]
at java.util.Iterator.forEachRemaining(Iterator.java:133) ~[?:?]
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) ~[?:?]
at net.minecraft.world.level.entity.PersistentEntitySectionManager.lambda$updateChunkStatus$6(PersistentEntitySectionManager.java:179) ~[patched_1.17.1.jar:git-Paper-186]
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) ~[?:?]
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179) ~[?:?]
at java.util.stream.LongPipeline$1$1.accept(LongPipeline.java:177) ~[?:?]
at java.util.PrimitiveIterator$OfLong.forEachRemaining(PrimitiveIterator.java:189) ~[?:?]
at java.util.Spliterators$LongIteratorSpliterator.forEachRemaining(Spliterators.java:2002) ~[?:?]
at java.util.Spliterator$OfLong.forEachRemaining(Spliterator.java:763) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) ~[?:?]
at net.minecraft.world.level.entity.PersistentEntitySectionManager.updateChunkStatus(PersistentEntitySectionManager.java:164) ~[patched_1.17.1.jar:git-Paper-186]
at net.minecraft.world.level.entity.PersistentEntitySectionManager.updateChunkStatus(PersistentEntitySectionManager.java:150) ~[patched_1.17.1.jar:git-Paper-186]
at net.minecraft.server.level.ChunkMap.onFullChunkStatusChange(ChunkMap.java:2112) ~[patched_1.17.1.jar:git-Paper-186]
at net.minecraft.server.level.ChunkHolder.demoteFullChunk(ChunkHolder.java:490) ~[app:?]
at net.minecraft.server.level.ChunkHolder.updateFutures(ChunkHolder.java:639) ~[app:?]
at net.minecraft.server.level.DistanceManager.runAllUpdates(DistanceManager.java:127) ~[app:?]
at net.minecraft.server.level.ServerChunkCache.runDistanceManagerUpdates(ServerChunkCache.java:640) ~[app:?]
at net.minecraft.server.level.ServerChunkCache.tick(ServerChunkCache.java:722) ~[app:?]
at net.minecraft.server.level.ServerLevel.tick(ServerLevel.java:555) ~[app:?]
at net.minecraft.server.MinecraftServer.tickChildren(MinecraftServer.java:1553) ~[patched_1.17.1.jar:git-Paper-186]
at net.minecraft.server.dedicated.DedicatedServer.tickChildren(DedicatedServer.java:479) ~[patched_1.17.1.jar:git-Paper-186]
at net.minecraft.server.MinecraftServer.tickServer(MinecraftServer.java:1406) ~[patched_1.17.1.jar:git-Paper-186]
at net.minecraft.server.MinecraftServer.runServer(MinecraftServer.java:1188) ~[patched_1.17.1.jar:git-Paper-186]
... 2 more
[18:50:19 ERROR]: Cause of unexpected exception was
java.util.ConcurrentModificationException: null
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1013) ~[?:?]
at java.util.ArrayList$Itr.next(ArrayList.java:967) ~[?:?]
at com.google.common.collect.Iterators$3.next(Iterators.java:174) ~[patched_1.17.1.jar:git-Paper-186]
at java.util.Iterator.forEachRemaining(Iterator.java:133) ~[?:?]
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) ~[?:?]
at net.minecraft.world.level.entity.PersistentEntitySectionManager.lambda$updateChunkStatus$6(PersistentEntitySectionManager.java:179) ~[patched_1.17.1.jar:git-Paper-186]
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) ~[?:?]
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179) ~[?:?]
at java.util.stream.LongPipeline$1$1.accept(LongPipeline.java:177) ~[?:?]
at java.util.PrimitiveIterator$OfLong.forEachRemaining(PrimitiveIterator.java:189) ~[?:?]
at java.util.Spliterators$LongIteratorSpliterator.forEachRemaining(Spliterators.java:2002) ~[?:?]
at java.util.Spliterator$OfLong.forEachRemaining(Spliterator.java:763) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) ~[?:?]
at net.minecraft.world.level.entity.PersistentEntitySectionManager.updateChunkStatus(PersistentEntitySectionManager.java:164) ~[patched_1.17.1.jar:git-Paper-186]
at net.minecraft.world.level.entity.PersistentEntitySectionManager.updateChunkStatus(PersistentEntitySectionManager.java:150) ~[patched_1.17.1.jar:git-Paper-186]
at net.minecraft.server.level.ChunkMap.onFullChunkStatusChange(ChunkMap.java:2112) ~[patched_1.17.1.jar:git-Paper-186]
at net.minecraft.server.level.ChunkHolder.demoteFullChunk(ChunkHolder.java:490) ~[app:?]
at net.minecraft.server.level.ChunkHolder.updateFutures(ChunkHolder.java:639) ~[app:?]
at net.minecraft.server.level.DistanceManager.runAllUpdates(DistanceManager.java:127) ~[app:?]
at net.minecraft.server.level.ServerChunkCache.runDistanceManagerUpdates(ServerChunkCache.java:640) ~[app:?]
at net.minecraft.server.level.ServerChunkCache.tick(ServerChunkCache.java:722) ~[app:?]
at net.minecraft.server.level.ServerLevel.tick(ServerLevel.java:555) ~[app:?]
at net.minecraft.server.MinecraftServer.tickChildren(MinecraftServer.java:1553) ~[patched_1.17.1.jar:git-Paper-186]
at net.minecraft.server.dedicated.DedicatedServer.tickChildren(DedicatedServer.java:479) ~[patched_1.17.1.jar:git-Paper-186]
at net.minecraft.server.MinecraftServer.tickServer(MinecraftServer.java:1406) ~[patched_1.17.1.jar:git-Paper-186]
at net.minecraft.server.MinecraftServer.runServer(MinecraftServer.java:1188) ~[patched_1.17.1.jar:git-Paper-186]
at net.minecraft.server.MinecraftServer.lambda$spin$0(MinecraftServer.java:316) ~[patched_1.17.1.jar:git-Paper-186]
at java.lang.Thread.run(Thread.java:831) [?:?]
[18:50:19 ERROR]: This crash report has been saved to: /minecraft/./crash-reports/crash-2021-08-20_18.50.19-server.txt
Plugin list
Agrum-Creator, ChestShop, CommandBook, CraftBook, LogBlock, LogBlockQuestioner*, LuckPerms, LWC, MCMailBox, Minequery*, MobArena, Multiverse-Core, Pitfall, ProtocolLib, spark, Vault, VoidGen, WorldEdit, WorldGuard, Ziggy
Actions to reproduce (if known)
No response
Paper version
This server is running Paper version git-Paper-186 (MC: 1.17.1) (Implementing API version 1.17.1-R0.1-SNAPSHOT) (Git: 72ce684)
Agreements
- [X] I am running the latest version of Paper available from https://papermc.io/downloads.
- [X] I have searched for and ensured there isn't already an open issue regarding this.
- [X] My version of Minecraft is supported by Paper.
Other
Not using the latest version of paper, but don't see anything that would have been related to the crash in the builds since. Also a sporadic crash so figured it should still be reported.
system specs are 100% useless, something, likely a plugin, looks to have modified the servers internal entity lists improperly
system specs are 100% useless, something, likely a plugin, looks to have modified the servers internal entity lists improperly
Do you have any ideas what kind of improper modification might have caused this? Since you're suspect of a plugin and most of the plugins are either mine or pretty popular ones, I should probably look for whatever it is...
system specs are 100% useless, something, likely a plugin, looks to have modified the servers internal entity lists improperly
Do you have any ideas what kind of improper modification might have caused this? Since you're suspect of a plugin and most of the plugins are either mine or pretty popular ones, I should probably look for whatever it is...
Do a binary plugin search
It's an extremely rare issue maybe 1/5 days... Narrowing it down to a plugin isn't all that helpful with thousands of lines of code in many plugins.
Try to figure out what happen right before the crash. Perhaps some kind of command is ran, check the logs for those kinds of things It being rare means the code that causes it is rarely ran, that way you can probably rule out a lot of things as the cause.
Sometimes it just happens at startup. I mean it's really random.
Hmm... Maybe it's some race condition in a player join handler...
Well, then start by checking your OnEnable, onLoad, perhaps even onJoin listeners?
You can also to begin with, start removing plugins in groups, perhaps 5 at a time, then you at least know it is one of 5, and then you can search further. Or perhaps first remove all of your own plugins?
Stack trace looks like #5998,
which cause by removing entities in event handler.
not sure only EntityRemoveFromWorldEvent or there are more?
Maybe can give a try.
It's anecdotal because it's so rare, and I haven't looked at the patch to see what it actually did/this makes any sense... But I don't believe we've experienced this since https://github.com/PaperMC/Paper/pull/6488 was applied to our server.
some of the seperations caused in that patch might have helped this given that the raw underlying state isn't as exposed, think one issue is that CB is removing entities in a way which isn't ultra friendly especially with the modern entity system
Well, the way entities are removed in general is not ultra friendly with the modern entity system... (This isn't CB specific, looks like that one was addressed in the wider set of changes to the entity system)
Vanilla works on the expectations that stuff happens in an ordered fashion, e.g. entity deaths will generally only occur and cause removal during the ticking of entity related stuff, the issue here is that you're attempting to remove an entity when the server is already removing entities and running code in an area which it doesn't expect an entity to be removed for any reason;
There are some cases we can take advantage of here, e.g. the entity will be marked as invalid, so, we can just set the thing as disposed and not run the removal logic, but, that may have some side effects, e.g. any passengers will be explicitly destroyed which may not be expected but is the only real course of avenue I can see here which doesn't involve a clusterfuck of moving logic around, etc
This is definitely still happening in 1.18, happened when two people went through a portal at the same time (though I doubt that's the cause) and another person was just walking around.
Alright, I have realized where this is happening... Though not the exact circumstance.
There's something that causes Minecraft to remove certain entities I'm using as bosses, so I wrote a system to catch that and replace the bosses on the fly.
The code is effectively:
@EventHandler
public void onEntityRemove(EntityRemoveFromWorldEvent event) {
Entity entity = event.getEntity();
if (!(entity instanceof LivingEntity)) {
return;
}
LivingEntity living = (LivingEntity) entity;
if (bossIsDead(living)) {
spawnBossEntity();
}
Which would add an entity while we're removing entities in pretty much the exact same spot, which increases the chances of the entitiy being on the same collection of entities for the chunk.
I should've put this together sooner.
Perhaps it would make sense to make these calls invalid while the list is being updated? e.g. set a flag that causes an exception if a remove or spawn is called during list traversal?
This is definitely still happening in 1.18, happened when two people went through a portal at the same time (though I doubt that's the cause) and another person was just walking around.
Could you post an updated stacktrace that applies to 1.18? I'm experiencing a very similar issue with a near-identical stack trace, but the line number and lamba number is different.
Indeed happens rarely, in my case on a server with about 30 players during peak hours, maybe once or twice a day.
my stack trace on git-Paper-"30cb7d0" (MC: 1.18.1) is as follows:
java.util.ConcurrentModificationException
at java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1013)
at java.base/java.util.ArrayList$Itr.next(ArrayList.java:967)
at com.google.common.collect.Iterators$1.next(Iterators.java:146)
at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
at net.minecraft.world.level.entity.PersistentEntitySectionManager.lambda$updateChunkStatus$7(PersistentEntitySectionManager.java:273)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
at java.base/java.util.stream.LongPipeline$1$1.accept(LongPipeline.java:177)
at java.base/java.util.PrimitiveIterator$OfLong.forEachRemaining(PrimitiveIterator.java:189)
at java.base/java.util.Spliterators$LongIteratorSpliterator.forEachRemaining(Spliterators.java:2002)
at java.base/java.util.Spliterator$OfLong.forEachRemaining(Spliterator.java:763)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
at net.minecraft.world.level.entity.PersistentEntitySectionManager.updateChunkStatus(PersistentEntitySectionManager.java:257)
at net.minecraft.world.level.entity.PersistentEntitySectionManager.updateChunkStatus(PersistentEntitySectionManager.java:241)
at net.minecraft.server.level.ChunkTracker.onFullChunkStatusChange(ChunkTracker.java:2164)
at net.minecraft.server.level.ChunkHolder.demoteFullChunk(ChunkHolder.java:517)
at net.minecraft.server.level.ChunkHolder.updateFutures(ChunkHolder.java:701)
at net.minecraft.server.level.DistanceManager.runAllUpdates(DistanceManager.java:245)
at net.minecraft.server.level.ServerChunkCache.runDistanceManagerUpdates(ServerChunkCache.java:839)
at net.minecraft.server.level.ServerChunkCache.tick(ServerChunkCache.java:913)
at net.minecraft.server.level.ServerLevel.tick(ServerLevel.java:625)
at net.minecraft.server.MinecraftServer.tickChildren(MinecraftServer.java:1617)
at net.minecraft.server.dedicated.DedicatedServer.tickChildren(DedicatedServer.java:480)
at net.minecraft.server.MinecraftServer.tickServer(MinecraftServer.java:1480)
at net.minecraft.server.MinecraftServer.runServer(MinecraftServer.java:1273)
at net.minecraft.server.MinecraftServer.lambda$spin$0(MinecraftServer.java:317)
at java.base/java.lang.Thread.run(Thread.java:831)
@martijn-heil do you write your own plugins or are you using only community plugins?
Our server hasn't been very active so I can't say for sure if this fixed it, but I have not seen the crash since updating my EntityRemoveFromWorldEvents to do any entity adding through the scheduler.
Both, we have a little team maintaining various custom plugins.
I still need to scan all our own code for any problematic usage of EntityRemoveFromWorldEvent like illustrated above.
Currently I'm writing up a little plugin to list all plugins that listen for the EntityRemoveFromWorldEvent and then I'm gonna work from that list.
I totally forgot to report my results.. but at long last: it turned out that the only plugin listening for this event in my case was MCPets. The problem was already solved in a next version of MCPets, we updated MCPets and we haven't experienced this ever since.
This issue has been expanded with the recode of the chunk system, as this issue now occurs when trying to move an entity as well. New Deleting Entity Exception: https://pastebin.com/raw/HBWfmAyH
New Teleportation Exception: https://pastebin.com/raw/53GUDLay
