RobustToolbox icon indicating copy to clipboard operation
RobustToolbox copied to clipboard

Integration tests are very leaky

Open perryprog opened this issue 5 months ago • 6 comments

Not sure when this was introduced, but we're running into this downstream as it appears to be hitting the Github Actions OOM killer of ≈ 10 GB so presumably Content isn't far behind from hitting the same problem. I'm not smart enough to diagnose the problem, but at the least it doesn't seem to be from the last engine update. It's also possible this is a long-standing issue that was just never noticed since memory usage in tests is relatively low priority.

Here's what I get when profiling space-wizards/space-station-14@1b43f6efd41188c88a4744c87024943224b811c2: Image

perryprog avatar Jul 23 '25 22:07 perryprog

Looks like RMC is running into the same issue. RMC-14/RMC-14#6218

perryprog avatar Jul 23 '25 23:07 perryprog

What's allocated?

metalgearsloth avatar Jul 24 '25 00:07 metalgearsloth

It seems like it's almost all entity prototype dictionaries on the client. (I think.)

perryprog avatar Jul 24 '25 00:07 perryprog

You have dotTrace open, get it to show a retention path for the relevant data.

PJB3005 avatar Jul 24 '25 13:07 PJB3005

Here you are:

Image

perryprog avatar Jul 24 '25 14:07 perryprog

I think I got red herring'd as actually comparing snapshots shows these are the sources of the new bytes:

Image

The B2 trees are retained from:

Image

It also looks like what I'm able to profile myself on macOS is somewhat limited as it seems only profiling memory via process attach for unit tests seems to work, so I can't use full allocation mode or get some of the call tree. (This is apparently skipping 300 MB of objects according to an out of screenshot warning.)

Image

I'm not sure if this is actually relevant, but I don't really understand how sampled memory allocation profiling works so I'm including it anyway. Presumably EntityTest is only a contributing factor regardless since disabling it shows a similar ramp of memory usage, which makes sense.

perryprog avatar Jul 24 '25 16:07 perryprog