Bandwidth estimation (possible rough ETA? Use-case feedback)

[Frooxius]

Hello!

I've recently got the Steamworks version of the library integrated with my project thanks to your additions (like the flat interface for C#)!

It's mostly working great, it's a lot more robust/efficient than our other protocol, except for one thing: the fixed bandwidth.

I know this is on the roadmap, so I just want to share my experience and add my +1 for this feature to be re-implemented. If you have any rough ETA on when it might come back that would also help, but I of course understand if you do not!

---

Our use-case:

Anyway, our project has a bit of varied network requirements. Initial connection can transfer a lot of data (hundreds of kB to MB) due to initial state sync (it's a VR world-building, so the entire data model gets synced).

During the session itself there are usually small deltas (using reliable channel) and position/voice and other stream data (sent as unreliable). Additionally there can be P2P asset transfers, which can be dozens, sometimes even hundreds of MB in addition.

For the assets, I technically establish a second independent connection (+1 for the ENet like channels, but it's workable without them https://github.com/ValveSoftware/GameNetworkingSockets/issues/95 ) so it's better interweaved with the realtime traffic (both reliable and unreliable).

However when using this protocol, the initial sync and asset transfers either end up being very slow when the bandwidth is conservative (the default seems to be about ~128 kB/s). Everything else is smooth.

If I up the bandwidth to something higher, like 8 MB/s, if the connection can handle it it also runs really well and transfers are super speedy. However once there are multiple people or people with slower upload/download, this ends up very quickly degrading everything.

The connection quality starts dropping to 0.5, even as low as 0.1, I assume because of a lot of dropped UDP packets, because the library is pushing them through too fast and it becomes unusable.

I've tried setting the limit just a bit higher, but even something like 512 kB/s causes problems, so we're currently stuck with the low default limit.

In our case we don't currently care about not sending some message, so if there was a limitation that bandwidth estimation only works well when the messages are not removed from the queue, that would be ok, but I understand if it's necessary to have mechanism like this in general anyway.

Currently we're staying with the old protocol as our primary one. It uses sliding window algorithm, so it scales better with the bandwidth, but it's not nearly as good as this one otherwise, we'd love to switch to it as our primary transport protocol.

I hope this info helps a bit! Thanks for all your hard work on this!

[zpostfacto]

I think your situation of a big blast at the beginning of a session, mostly small incremental updates, with occasional large chunks, is very typical. For delivery of large assets....if possible at all I really recommend using some other mechanism to distribute those, like hosting them on a Web server and sharing the URL. Even if it is the same host serving game traffic and assets! I know this places an extra configuration demand on the game server operator, but it really is vastly superior, and it enables people to take advantage of all of this engineering effort and infrastructure, such as intermediate web caches. Hopefully you can make it so that the assets are only transferred over the game channel as a backstop / last resort. But then at least players have a known, well-understood mechanism available where they can optimize it on their own.

I'm hopefully going to have NAT piercing finished this week. I have it working, but now remaining is to strip back WebRTC layers and figure out how to opensource our thin interface layer to it so that people could build or integrate WebRTC how they want. WebRTC is massive. Also, I will make it possible to use other ICE / SIP libraries, but I'll have the WebRTC integration as a template.

After that, bandwidth estimation is next up. Many of my requirements are currently being driven by steamlink and remote play together, which is a customer of this library. The bandwidth estimation in steamlink essentially uses latency decrease as the primary signal for bandwidth nearing capacity. This works really well on a Wifi network, but it is not how the Internet responds to congestion. So....hopefully I'll be starting on BBR soon. This won't be as simple as just integrating some google BBR code, because I don't have a sliding window, and I have unreliable data. It just doesn't map exactly onto available code.

So all that is a rambling way to say....my timeline is.....hopefully soon?

[Frooxius]

Oooh that's awesome, thank you so much for the update! I didn't know that NAT piercing would come so soon (I assumed it would still be months at least), that will help us a lot as well.

And definitely, serving assets from a webserver is superior, most assets with our project are served this way, but it requires users to save them into their inventory (cloud storage) first.

We use the game connection as a fallback for assets that are created/imported during the session to sync them up between the users without having to push them to the cloud first. It does some clever things with certain assets like textures, by streaming the mipmaps over the connection as they're still generated on the user who created/imported them, which significantly lowers latency from the user importing/creating the asset to a first (blurry) version appearing to other users.

Thanks again for the update and all the work on this library! I appreciate the info, even if it takes longer in the end.

[shiftyscales]

Hey @fletcherdvalve- any chance you might have an updated timeline/estimation for when bandwidth estimation may get integrated?

Thanks for the work you've done on this project.

[pipliz]

My use case is similar to the one from OP - loading world state (voxels) of say 200 KiB to 3 MiB initially, and then typical gameplay is a smaller continious rate - except for situations like teleporting where you get the initial state loading spike again. Currently I'm making do with TCP and the 'legacy' steam networking. What I really need is more a "reliable, fragmented UDP" version. This library seems ideal as it provides that and it's one of the few ones where you can read the estimated send buffer delay, the estimated bandwidth, transmission quality / drop rates and such. With that data, I can manually throttle the further away voxel data being sent too quickly and thus prevent overloading the connection, keeping priority data ping happy.

I tried the version in the steam SDK a bit over a year ago, so it's quite possible that my experience is outdated. The default estimate was really low the moment some ping was involved - impacting the initial loading too much at say 100 KiB/s while the connection supports 3 MiB/s. I added some manual bandwidth estimation and it worked fine for local connections. Once I used it for a real connection with practical limits, it would work fine until the estimate went over an actual real bottleneck like ISP upload bandwidth. At that point the connection quality would instantly drop to 0 and wouldn't recover even if the ratelimit was set back lower below any limits. I'm not sure what was going on there - maybe I should've made my own issue at the time but hey.

Once this feature is in and propagated to the steam SDK I'll likely test the library again

[Stellanora64]

@zpostfacto are there any updates on the progress of this?

[Frooxius]

I'd like to say that we're still interested in this feature being implemented. It's still a blocker for us switching to this library as our main transport - we are currently using a library that's inferior in a lot of ways, but better adapts to user's available bandwidth (it uses a sliding window), which behaves better overall.

We have also begun looking into other alternatives as well, in case this falls through, but we ideally like to use this one if possible.