Extreme loading times (with Sky Islands)

[Terrorforge]

Describe the bug

I've been playing Sky Islands on the same save file since it came out in the middle of January. My load times have been steadily increasing the whole time, and have now become patently absurd. I'm about 90 days in, the save file is just under 3gb in size, and it takes over half an hour to load; 34 minutes, last I checked. This is on an SSD.

I tried deleting some map files, bringing the total size down to about 300mb, but the load times did not improve. Even after trimming and compressing with the debug function, load times did not improve.

Attach save file

Skypeia-trimmed.tar.gz CDDA-Sky-Islands-main.zip

The second file is the version of Sky Islands I'm using, which is the latest one at time of writing.

Steps to reproduce

1. Unzip the Sky Island files into your mod folder
2. Attempt to load my save

(Do this on an SSD if you want to actually get into the game within a human lifetime)

Expected behavior

Load times of less than half an hour. Ideally a lot less.

Screenshots

No response

Versions and configuration

• OS: Windows
  • OS Version: 10.0.19045.2604 (22H2)
• Game Version: d6ec466 [64-bit]
• Graphics Version: Tiles
• Game Language: English [en]
• Mods loaded: [ Dark Days Ahead [dda], Disable NPC Needs [no_npc_food], Portal Storms Ignore NPCs [personal_portal_storms], No Fungal Growth [no_fungal_growth], Bionic Professions [package_bionic_professions], Sky Island [skyisland] ]

Additional context

This could obviously be a bug in Sky Islands rather than the main game, but it might be a symptom of a more fundamental issue so @harakka asked me to create an issue here.

Also I'm using Catapult in case that matters.

[andrei8l]

/confirmed The achievement tracker goes off the rails with allocations due to achievement_return_to_first_omt. Any other achievement with tons of events would cause the same issue.

[Terrorforge]

Is that something that can fixed in short term by e.g. deleting the relevant part of the save file?

[andrei8l]

Yeah, just delete "achievement_return_to_first_omt", from file #SmVyaWNobw==.sav

[Terrorforge]

Yep, did that and I can confirm that it now loads in less than a minute. Amazing.

[akrieger]

I've been itching to tackle stats tracker's O(n) serialization size. Achievement tracker having the same issue is even more motivation.

[anoobindisguise]

#62591 is highly relevant here. I got up to over an hour load times on a separate occasion and then deleted "avatar_enters_omt" and my load times were instant.

[akrieger]

@jbytheway I've tried to tackle this twice by updating stats_tracker to not need to keep around each individual unique event it reads off the event_bus, because that's a linear growth over the length of a game, but I've failed both times. Even if that would work for avatar_enters_omt, it breaks other things. I kicked around ideas like explicitly marking events that do/don't need to be uniquely tracked, but that feels icky. Any thoughts here?

[jbytheway]

I don't believe this is just an issue with linear growth of the avatar_enters_omt. In 34 minutes we ought to be able to process many millions of OMT entry records if it was just linear growth. I believe there's a quadratic runtime issue in there somewhere.

I vaguely recall about a year ago I tracked down such an issue and started trying to fix it, but got distracted and never finished. Now that it's impacting users to this degree that gives me the motivation to revisit it. I'll see what I can figure out.

[jbytheway]

Took me a while, but I've finally tracked down the TODO where I had explained to myself how to fix this, so hopefully I can tackle this issue soon.

[jbytheway]

Putative fix at #64844. Your example save now loads in effectively no time (negligible compared to the time for loading the game data).