FakeFishGames
(0.18.7) Pirate sub brutally killed performance? (event count very high )
Disclaimers
- [X] I have searched the issue tracker to check if the issue has already been reported.
- [X] My issue happened while using mods.
What happened?
Multiplayer campaign route with a pirate- and swarm-mission grinded our dedicated server (6 clients total) to a halt for almost 5mins.
No technical bottlenecks monitored. Core- (~20% load), ram- and network-traffic (~220-300kbit/100mbit total) was low and had lots of headroom)
After a couple of reloads we managed to re-dock and skip the mission. Performance was instantly back to normal. This was the first time in this campaign that performance was this brutally culled. We havent even had this degree of issues on a route with 3 swarm missions (mob spawns still collapsed performance on our lower end clients).
Suspected: The updates of the pirate vessel (and its crew) clogged the servers network layer without clogging actual network straffic. Maybe in addition to the swarm spawns, which cause regular performance collapses on some clients.
Only mod used: New Wrecks For Barotrauma (With sellable wrecks) Custom sub
Reproduction steps
- We run the campaign as dedicated server. (AMD Ryzen 3600)
- 6 players MP
- Server was up maybe 1.5-2hrs
- CPU, RAM and Network load was neglectable (lots of headroom, no significant packet losses, ping to clients 40-65ms)
- Start of the mission tanked the game performance to a halt for almost 5mins.
- Reloads of the campaign had the same results (saves attached)
- server threw lots of error messages (attached pic)
Marburg0.17_Mai22 - Kopie.save.zip
Bug prevalence
Happens every time I play
Version
0.18.7
Which operating system did you encounter this bug on?
Windows
Relevant error messages and crash reports
No response
I confirm. Starting from 18.7 the pirate submarine is very laggy. Obviously more than before.
Tried to diagnose this by switching levels rapidly in multiplayer campaign, although not specifically the same I think the cause is similar (I had yet to encounter any pirate missions or pirates unfortunately).
I did encounter a singular wreck corpse causing an excessive character spawn data (Though 256 bytes vs 255 max) just barely over the max.
what I can confirm is this was caused by the corpse of a wreck's doctor. the wreck had only 3 corpses generate (So 1/3 caused this and I probably passed many wrecks and corpses without seeing this once).
I did try to back out of the server...but it did not get past connecting or reach lobby despite the round was working fine enough that I hadn't noticed it and was seconds from reaching the next destination by sub teleports to try to pickup more missions rapidly with other missions on the side.
My theories on this may be related to the amount of items they start with (?) perhaps, or some other aspect of data that could bloat related to wreck and pirate generation.
Unfortunately, loading the same save did not generate a wreck going the same path with no missions (Which I had before).
the only place I experienced event counts being very high was a singular outpost.
Event count did report being high at 360, Another attempt generated a high event count of 389 (122 spawner, 83 levelobjectmanager and 39 barotrauma.hull) with another wreck of 6 corpses - no character spawn data errors occurred further note it was the same wreck that generated.
Caught this once and got a little more specific info:
Since I cannot reliably reproduce this via save (The wrecks keep changing and pirate submarine missions don't generate much and are always not the elite type until I got to a high enough difficulty that no outposts existed to spawn missions for them).
message length ended at 178 bytes + 80 bytes at attempting to write status at the second error (So 178 bytes before writestatus and +80 equaling 258 bytes afterwards)
InitialMsgLength = 2 bytes, msgLengthBeforeInfo = 43 bytes, Info length = 135 bytes, ordersLength = 1 bytes.
writestatus had a total of 11 valid (Value not null) afflictions to send. Each affliction uses a UIntIdentifier (4 bytes of data) so: 1 byte bool for dead/alive 4 bytes for cause of death int (If cause of death is affliction, write affliction identifier on top) 11*4 = 44 bytes, might be worth it if afflictions specifically could use a different identifier even if just for networking purposes only (even 1 unsigned byte would give you some 256 options for afflictions, unsigned 2 byte would give you 32767 max affliction prefabs to send an identifier of).
Small none-issue worth consideration: skills are written for wreck corpses too, using string identifiers (I bet that costs more bytes than necessary when we typically only have 5 or 6 skills and I can't picture modding in skills going past 15 but this alone probably bloats the info considerably as we send full strings)
Either way, there are probably many ways to reasonably reduce the byte usage of these messages making them smaller and fitting more events per packet.
I believe pirates also cause this issue based on the first, but given the orders for the first occurrence in the git issue have an orders size of around 26 for some yet an info of 169+ bytes.
As for the lag on too many events cascading into more events potentially than can be sent per second (That is what this issue feels like) I don't think this is the direct cause (Or it would never stop making errors on it if it never stopped sending them) but is a potential contributor (Less events per packet from complex characters like pirates having larger events to send), the issue needs to be kept open in case we can encounter that (And its most likely tied to the highest difficulty elite pirate vessels), may figure out a way to test this later through modding event difficulties to get the elite pirate ships to spawn to see if its possible to reproduce.
I was able to get similar results by giving the wreck corpses excessive amounts of afflictions (by modifying Level.SpawnCorpses so it gives the corpses all the afflictions that can potentially be given by that method). Addressed in https://github.com/Regalis11/Barotrauma-development/commit/1229950f8b70d56d6175d1b25a8f7c11c8a92da8, although I'm not sure this is enough to completely solve the issue.
Tested against dev commit https://github.com/Regalis11/Barotrauma-development/commit/f82a333e436d97a8051448f960194e41892e9be5
Tested on the provided save file for the performance using the same wreck mod.
The fixes have definitely seemed to resolve (for the most part, at least with vanilla afflictions on wreck corpses) most if not all occurrences of writing character spawn data networking issues (but if we can optimize the networking on these, it may still be worth doing so at a later date, the round starts for writing events do take some time and its not a guaranteed fix as much as it likely makes it an extreme edge case or modding issue now).
Those however were not the cause of performance issues. I have no reasonable means of measuring the performance. but ran the showperf command in a recording (with my hardware not taxed at all, such as 1-3% on my GPU load and plenty of cores to go around without maxing them and memory available. disk usage 0% and such. Each thread/core runs at 4.1Ghz for sake of reference as that appears to possibly be the bottleneck here and is mostly in the draw area on the CPU side (?).
Yes - I am running a debug build in this which uses unoptimized code, so there is a performance overhead in the visual studio debugging, but since its a single recording it is consistent within that debugging session.
Lights set to 100 max. Particles max count 1000.
Notes - I've had to compress the video to reduce file size from 53mb's to under 10mb's. Numbers are still visible, but recording quality and/or framerate may have been reduced: https://user-images.githubusercontent.com/29177976/186127354-eaa6f2e0-2da7-4f84-af1a-3e63ed4c1b60.mp4
Based on the video, it seems like this is an issue with rendering performance caused by the sheer number of structures and items it's rendering, which we have existing tickets about (https://github.com/Regalis11/Barotrauma-development/issues/2707, https://github.com/Regalis11/Barotrauma/issues/9396), so I think this one can be closed.