chunkgen
chunkgen copied to clipboard
gecgooden
Chunkgen large areas leads to Out-Of-Memory exceptions.
Chunkgen seems to loads chunks as quickly as it can with no apparent limiter. If forge doesn't unload chunks faster than chunkgen is loading them the server eventually dies. The chunk generator says unloadChunksIfNotNearSpawn but that doesn't seem to be as fast as chunkgen is loading them. Ram for the minecraft server is 2.5GB and total ram in the server is 8gb. RAM normally isn't ever an issue unless I use chunkgen on a large area. Would it be possible to check for the number of loaded chunks when starting gen then pausing the gen for a bit if the loaded chunks get say 250 more than that initial value? Or perhaps add in a manual save-all every so many chunks to help convince forge to unload the chunks?
/chunkgen 0 0 100 100 0
--SNIP--
[01:29:51] [Server thread/INFO] [chunkgen/]: percentage: 0.69200003
[01:29:51] [Server thread/INFO] [chunkgen/]: Loaded Chunk at 19 -30 0
[01:29:51] [Server thread/ERROR] [FML/]: Exception caught during firing event cpw.mods.fml.common.gameevent.TickEvent$ServerTickEvent@55a88681:
java.lang.OutOfMemoryError: GC overhead limit exceeded
This becomes even more apparent when /chunkgen 0 0 100 100 0 crashes then after starting back up you repeat the command only to have it crash even faster.
Can you try again with 1.3.0 from the releases? I've just merged in a bunch of commits from @Baughn which may improve this.
I generated the same world and seed twice (same one that crashed every time before) with v1.3.0. Doing a full world wipe and server restart each time for all my testing. No OOM exceptions and believe it or not the gen was nearly 2x faster. 10 minutes to generate the same command in my OP. Before It was taking 10 minutes to get a little past half way then OOM crashing. I reverted to v1.2.3 and the crashing came back at roughly the same place as before so definitely not a fluke.
/chunkgen 0 0 100 100 0
[19:58:23] [Server thread/INFO] [chunkgen/]: Loaded Chunk at 1 1 0
--SNIP--
[20:07:24] [Server thread/INFO] [chunkgen/]: Loaded Chunk at -49 50 0
[20:07:24] [Server thread/INFO] [chunkgen/]: percentage: 1.0
Looking at the start and end chunks I think there is a off by one error some where but it is largely insignificant to the original issue. I'll take the off by one in this case. :D
@Baughn Did good work!
I'm pretty sure it's still doing the same thing. I had chunkgen running overnight, and the server's basically frozen.
Look at https://goo.gl/kEeRB7. It's pretty obvious where I started chunk-genning. (Unfortunately I'm not tracking number of loaded chunks in prometheus-integration yet.)
It didn't OOM, and was eventually able to exit, but that may just have been my GC parameters.
The flat bit shortly after the initial spike corresponds to when I was actively playing, so chunkgen was paused. Unfortunately, it appears that simply giving the server time doesn't make it recover.
I'm trying to fix the generation of large area. So far, I think i am able to do so by unloading each chunk after it's generated instead of dropping all chunk after the whole thing is done. So far, i was succesfully able to generate 400 by 400 of ungenerated chunk without crashing the server. I'll do more test before proposing a pull request.
Just a thought, but it may be more efficient to, say, unload chunks every 256 chunks generated.
I don't actually know this. Time for since tests!
In my experience in doing large chunkgen back in ... 164? 1710? Don't recall, the most important thing to do was to switch the garbage collector back to throughput (the normal parallel pause the world collector). While CMS made playing much better, it had horrible performance doing garbage collection as space got filled up, while the parallel collector gave great performance all the way to 98% full.
... I think (from memory) I was doing an r=50 generation.
I'm currently trying with 1875 by 1875 for a 30000 x 30000 blocks and it's going good so far. We can unload all chunks or unload chunks by coordinate. Generate then unload seems good to me.
On Thu, Apr 28, 2016, 2:18 PM keybounce [email protected] wrote:
In my experience in doing large chunkgen back in ... 164? 1710? Don't recall, the most important thing to do was to switch the garbage collector back to throughput (the normal parallel pause the world collector). While CMS made playing much better, it had horrible performance doing garbage collection as space got filled up, while the parallel collector gave great performance all the way to 98% full.
... I think (from memory) I was doing an r=50 generation.
— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/gecgooden/chunkgen/issues/12#issuecomment-215516959
Ahh. Well, I had a 4 GB machine, about 3.something assigned to Java, and was doing a mystcraft/COG combo to try to determine how many ores COG was actually generating.