Very slow download when joining a modded server

[mariovata]

Disclaimers

• [X] I have searched the issue tracker to check if the issue has already been reported.
• [ ] My issue happened while using mods.

What happened?

(I am aware that I have not selected that my issue happened while using mods. I did this because I believe a core functionality of the game has a bug and is not caused by a mod, but rather having any mod)

I have hosted a server with mods and when someone tries to join it takes them a long time because the download seems throttled. I am not sure if this is on my end as I have a stable and fast connection (524.00 Mbps download, 260.77 Mbps upload). When I asked a friend to join, they said that it is a lot slower than before the update. Furthermore, when they downloaded the mods and reconnected, they had to download the same mods again. Unfortunately, subscribing to the mods on the steam workshop did not help, and they had to download the mods from the server again. Is there something I can do to fix this? I was wondering if I hosted a dedicated server, it might help with the download speeds.

Reproduction steps

1. Join any modded game.
2. Observe the download speed is much slower.

Bug prevalence

Happens every time I play

Version

0.17.15.0

-

No response

Which operating system did you encounter this bug on?

Windows

Relevant error messages and crash reports

No response

[Clintonio]

My friend and I are having the same issue. My upload speeds are confirmed to be >100Mb/s, I am connected via Ethernet, likely behind a carrier NAT. He is on a similar fast connection. This doesn't happen on other games, while streaming, or generally.

[Clintonio]

I tried without mods and the issue persisted.

[ClassyMan]

I am also experiencing this issue and have been for some time. The order of events is as follows:

• Save file download takes far too long
• The host gets put into the round ahead of me
• I have to therefore wait for the round to end before joining
• At which point I need to download a new save file and the cycle repeats

I feel like there are some suggestions here in terms of a fix:

• Fix the code that downloads the host's save file to be more performant
• Simply cache the save file so that the host can just restart the game, and from that point I already have the file locally, so can immediately join
• Allow users to pre-load a file before starting (this might be possible as a workaround so I will check, presumably files are stored in a folder somewhere and I can see if we can just send one another the files off-game)
• Have the game wait until all players have the same save before allowing players to start the game, or sync up in the same way at round ends. Possibly have this as a configurable setting for the host.

At present this is basically game-breaking for me since I only play multiplayer with friends.

[theCrius]

Checking today as I was messing with a modded server. The download from the server side ~~is~~ seems to be artificially capped at chunk of 1000 bytes (1KB). Each chunk need to be transferred and confirmed which, if you know a bit about transfer protocols, adds up very quickly and slow down the download quite significantly.

A settings server side to select the chunk size or just being capped at an higher chunk should alleviate the issue.

Edit: As mentioned by Regalis, protocol is not TCP so this theory goes out of the window I suppose

[Clintonio]

Checking today as I was messing with a modded server. The download from the server side is artificially capped at chunk of 1000 bytes (1KB). Each chunk need to be transferred and confirmed which, if you know a bit about transfer protocols, adds up very quickly and slow down the download quite significantly.

A settings server side to select the chunk size or just being capped at an higher chunk should alleviate the issue.

Yeah, that makes sense. Given the ping between my friend and I, (Tokyo to San Fran) this choice of networking will have a huge impact. Increasing that cap or allowing it to be configured would be great. That or switching to UDP, though that is considerably more work.

[WGOS]

Same issue. One guy from my party downloads mods/saves/subs very slow and this makes him dropped from server because of many reasons like "campaign map not loaded yet" or "The submarine at your end doesn't match the one loaded by the server" etc. Though other people can join server just ok.

We're already tried using different VPNs and different server locations. Both server have 1 gbit/s uplink and guy has 100 mbit/s uplink (confirmable by speedtest and also iperf3)

[WGOS]

https://github.com/Regalis11/Barotrauma/issues/9224#issuecomment-1159756518 Interestingly he has problem joining only dedicated servers but if server created using in-game lobby everything works fine

[Regalis11]

Thank you for the reports! Do you know which specific update this started appearing in? It seems strange to me that the transfers would've been earlier before, because transferring mods wasn't implemented until v0.17.15 (Urban Expanses update) and to my knowledge we haven't done any changes to the transfer logic since then.

Checking today as I was messing with a modded server. The download from the server side is artificially capped at chunk of 1000 bytes (1KB). Each chunk need to be transferred and confirmed which, if you know a bit about transfer protocols, adds up very quickly and slow down the download quite significantly. A settings server side to select the chunk size or just being capped at an higher chunk should alleviate the issue.

Yeah, that makes sense. Given the ping between my friend and I, (Tokyo to San Fran) this choice of networking will have a huge impact. Increasing that cap or allowing it to be configured would be great. That or switching to UDP, though that is considerably more work.

This isn't actually entirely accurate: the file transfers do use UDP, and the server doesn't wait for an acknowledgment before sending the next packet (at least until a certain point - if the client can't keep up, the server will start throttling the transfer and resending packets). Or at least this is how it is intended to work - it could be that we've accidentally broken something in the logic in the recent updates.

[theCrius]

Checking my docker, I can see that I was first running v0.17.15.0 but I can't tell if I noticed this issue already on that version. Unfortunately I only started adding mods to the server after a bit, probably 2-3 updates later.

If you want a quick spin-up environment you can check my repo barotrauma-docker.

About the protocol, you are right, didn't remember it was using UDP and that threw me off for a false lead :)

[Regalis11]

(Note: the links are to commits in our private repo, linking them here for our internal use)

I think this commit is the most likely cause to the reduced file transfer speeds, essentially cutting the rate at which the server can send packets in ideal conditions to 10%.

Running the Send method up to 10 times per Update was originally implemented in https://github.com/Regalis11/Barotrauma-development/commit/46037d5cff493502a0ebdab824db71a0efc59caa, and then a sort-of duplicate version of that logic with more robust congestion prevention was added in https://github.com/Regalis11/Barotrauma-development/commit/778290f2f03128e1da8d15d003c3cea00bfbba76. The latest commit removed the earlier logic without adjusting the new logic, which would result in the speed getting cut to 1/10.

[Regalis11]

@ClassyMan

Allow users to pre-load a file before starting

This is actually somewhat possible already, although when you load or start the campaign for the first time, the server tries to start the round immediately after selecting the save, so there's no time to pre-load. However, as a workaround before we manage to fix this properly, you might want to try re-joining the server if you fail to start the round. The server should send the save as soon as you join, and when you've received it, you can click "spectate" to join the round.

Have the game wait until all players have the same save before allowing players to start the game, or sync up in the same way at round ends. Possibly have this as a configurable setting for the host.

This is also implemented already, although at the moment the server only waits for up to 25 seconds, which seems to not be enough if the transfers are slow. I think we could definitely make it an adjustable setting (in addition to addressing the slow downloads ofc).

[theCrius]

Just adding a probably relative issue. It happens that people can join a match (in progress) with spectate but get kicked out for missing some resources. Can't recall the message now but I remember reading and thinking "I must haven't finished downloading the subs" because of course, when joining the server I already had all the mods but somehow subs get redownloaded every time. In case it help, subs for my server are put in the LocalMods folder (and listed in the config_player.xml). Couldn't find a way to have them downloaded by the client in any other way.

[Regalis11]

Addressed in https://github.com/Regalis11/Barotrauma-development/commit/e1f9a4a07f160de60d89ee7c9e8d89b036bda079

[mariovata]

I have found a workaround? The mods will not need to be downloaded if the server and the client:

1. Have the same mods
2. Have the mods placed in the same order

Hopefully this will help others as it is quite obvious, and I have just discovered this yesterday 😅. Unfortunately, this will only help when playing with friends, unless a server host publishes the mods and their order.

[ClassyMan]

Error thrown when attempting to start a match with another person with the same mods in the same order.

   at Barotrauma.MultiPlayerCampaign.ClientRead(IReadMessage msg) in <DEV>\Barotrauma\BarotraumaClient\ClientSource\GameSession\GameModes\MultiPlayerCampaign.cs:line 698
   at Barotrauma.Networking.GameClient.ReadLobbyUpdate(IReadMessage inc) in <DEV>\Barotrauma\BarotraumaClient\ClientSource\Networking\GameClient.cs:line 2213
   at Barotrauma.Networking.GameClient.ReadDataMessage(IReadMessage inc) in <DEV>\Barotrauma\BarotraumaClient\ClientSource\Networking\GameClient.cs:line 780
   at Barotrauma.Networking.SteamP2PClientPeer.Update(Single deltaTime) in <DEV>\Barotrauma\BarotraumaClient\ClientSource\Networking\Primitives\Peers\SteamP2PClientPeer.cs:line 240
   at Barotrauma.Networking.GameClient.Update(Single deltaTime) in <DEV>\Barotrauma\BarotraumaClient\ClientSource\Networking\GameClient.cs:line 634

------------------------------------------------------------------------------------------

Error while reading a message from server. {Sequence contains no matching element}
   at System.Linq.ThrowHelper.ThrowNoMatchException()
   at System.Linq.Enumerable.First[TSource](IEnumerable`1 source, Func`2 predicate)
   at Barotrauma.ModDownloadScreen.Select() in <DEV>\Barotrauma\BarotraumaClient\ClientSource\Screens\ModDownloadScreen.cs:line 84
   at Barotrauma.Networking.GameClient.OnConnectionInitializationComplete() in <DEV>\Barotrauma\BarotraumaClient\ClientSource\Networking\GameClient.cs:line 1278
   at Barotrauma.Networking.SteamP2PClientPeer.Update(Single deltaTime) in <DEV>\Barotrauma\BarotraumaClient\ClientSource\Networking\Primitives\Peers\SteamP2PClientPeer.cs:line 225
   at Barotrauma.Networking.GameClient.Update(Single deltaTime) in <DEV>\Barotrauma\BarotraumaClient\ClientSource\Networking\GameClient.cs:line 634
   
   ------------------------------------------------------------------------------------------

Error while reading a message from server. {Sequence contains no matching element}
   at System.Linq.ThrowHelper.ThrowNoMatchException()
   at System.Linq.Enumerable.First[TSource](IEnumerable`1 source, Func`2 predicate)
   at Barotrauma.ModDownloadScreen.Select() in <DEV>\Barotrauma\BarotraumaClient\ClientSource\Screens\ModDownloadScreen.cs:line 84
   at Barotrauma.Networking.GameClient.OnConnectionInitializationComplete() in <DEV>\Barotrauma\BarotraumaClient\ClientSource\Networking\GameClient.cs:line 1278
   at Barotrauma.Networking.SteamP2PClientPeer.Update(Single deltaTime) in <DEV>\Barotrauma\BarotraumaClient\ClientSource\Networking\Primitives\Peers\SteamP2PClientPeer.cs:line 225
   at Barotrauma.Networking.GameClient.Update(Single deltaTime) in <DEV>\Barotrauma\BarotraumaClient\ClientSource\Networking\GameClient.cs:line 634
   
   ------------------------------------------------------------------------------------------

Error while reading a message from server. {Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')}
   at Barotrauma.MultiPlayerCampaign.ClientRead(IReadMessage msg) in <DEV>\Barotrauma\BarotraumaClient\ClientSource\GameSession\GameModes\MultiPlayerCampaign.cs:line 698
   at Barotrauma.Networking.GameClient.ReadLobbyUpdate(IReadMessage inc) in <DEV>\Barotrauma\BarotraumaClient\ClientSource\Networking\GameClient.cs:line 2213
   at Barotrauma.Networking.GameClient.ReadDataMessage(IReadMessage inc) in <DEV>\Barotrauma\BarotraumaClient\ClientSource\Networking\GameClient.cs:line 780
   at Barotrauma.Networking.SteamP2PClientPeer.Update(Single deltaTime) in <DEV>\Barotrauma\BarotraumaClient\ClientSource\Networking\Primitives\Peers\SteamP2PClientPeer.cs:line 240
   at Barotrauma.Networking.GameClient.Update(Single deltaTime) in <DEV>\Barotrauma\BarotraumaClient\ClientSource\Networking\GameClient.cs:line 634```
   
   

[Rokvach]

Tested this with Nilanth, we didn't spot any issues while testing it in SteamP2P and dedicated and with various sizes of mods, the download speed was noticeable faster than previously. Can be merged to dev and closed.