Yellow-Dog-Man
Mesh Networking
Is your feature request related to a problem? Please describe.
Currently from what I understand, the host takes the burden of being the single point of distribution of all assets in a session. This makes it extremely expensive and does not scale well with many users in the session.
Mesh networking would be a major step towards optimizing network bandwidth away from the host by allowing individual users who already have downloaded the assets to distribute them amongst themselves (possibly based on their own estimated capacity to do so).
From an alternate dimension Froox:
This will heavily optimize network bandwidth usage, by interconnecting individual users and transferring some of the data between them directly (particularly position and voice streams and session asset chunks as well).
This will change the network bandwidth requirements on the host from roughly squared (every user joining doubling the bandwidth for the host) closer to linear (to double the bandwidth, twice as many users needs to join).
Describe the solution you'd like
The solution would be to implement some sort of mesh topology which can interconnect users within a session and negotiate the transfer of assets/other streaming data.
It would be possibly useful for whatever solution that is implemented to be able to estimate a user's capacity to send/receive, allowing the network traffic to be optimized.
Describe alternatives you've considered
The alternative is the current "Star Network" topology which is a single point of failure.
Additional Context
No response
The current situation results in $O(n^2)$ scaling. The proposed solution moves it to $O(n)$. It'll probably be twice the bandwidth at least on average per node, but it would greatly reduce the burden on the host.
To give an idea of how poorly the current architecture scales, one of the alternatives is to split a session into two equally-sized sessions. If both sessions are hosted on the same host, this would halve the network bandwidth required for the host.
I would argue that only doing this for asset transfer wouldn't be enough to fix everything - one would also need to do this for voice traffic and potentially avatar position stream information or whatever the heck happens there. However, it'll still be better than nothing, and it'll improve loading times if done right.
Ideally one would do everything through a mesh network for scalability.
On top of that, if everything is at least able to be fed through a mesh, then it opens up the possibility of not needing to go through the relay server(s) for some people in some cases.
Examples:
- Someone has a bad NAT, but at least one other person in the session does not. In that case, initial connections could be performed through those other people.
- One user is only on IPv4, one user is only on IPv6, and at least one user is on both. In that case, the users can still form a session.
Of course, if a specific set of users leave, then one would need to be able to quickly fall back on the relay server(s).
There are some security implications here, however. Asset transfers alone should be fairly straightforward as long as the server provides a cryptographic hash for the contents. World state updates on the other hand would probably need to be cryptographically-signed by the host, and user-provided streams would probably need to be cryptographically-signed by the relevant users.
This does have a CPU cost, so one might want to opt for something a bit lighter, or sign bundles of cryptographic hashes of voice packets to reduce the number of signatures required. With strong enough protection against accidental modification (CRC-32 is great for this and is blazingly fast - don't waste your time with FNV it's not worth it), you only need to detect one modified packet which passes the accidental modification check to know that you're dealing with a malicious node.
If one wants extra hashing speed, BLAKE3 has some nice characteristics.
Ultimately, having something now is better than waiting for everything to be ready later. Asset exchange should be the less difficult problem to solve. Of course, there would need to be some groundwork before any mesh network stuff can happen.
Just my pile of two cent coins. (Which of course haven't been legal tender for decades here.)
This is something that I've been considering for a while. The mesh networking would be used for asset transfers and streams - voice & pose data mainly. That usually represents majority of the heavy bandwidth in a session.
This would also help with other things too - users physically closer to each other would get better latency, as the traffic would not need to round trip to the host and it would also get around when the host hiccups, keeping the voice and pose data more stable.
The main issue here is determining which users are okay to connect to each other - not everyone likes their IP's being exchanged with every user in the session. Though we could use Steam Networking Sockets in this case, which do relay over the Valve network and hide the user's IP's from each other.
Not sure when this would get prioritized though, as it does require a good chunk of work.
Another issue when using mesh networking would be if you have an asymmetric connection. For example, I currently have 200 Mbit down and 20 Mbit up. I'm fine when connecting to a host with star topology, but I wouldn't be able to stream to many users myself in mesh.
UK networking is also a nightmare for this as well (almost every connection here is heavily asymmetrical). My previous connection was 22 down, 4 up. But trying to push anything through the upstream (not even getting near 4Mbit) would result in a completely crippled downstream. So the effect on a user's network really needs to be considered.
I currently have 200 Mbit down and 20 Mbit up.
*cries in German 60 down, 11 up*
Although hosting on that works fine usually, and even taking some strain off the host could do wonders
Yeah, we'll definitely want to consider options on this, so users with low upload are not strained. That will definitely be considered.
Another issue when using mesh networking would be if you have an asymmetric connection. For example, I currently have 200 Mbit down and 20 Mbit up. I'm fine when connecting to a host with star topology, but I wouldn't be able to stream to many users myself in mesh.
On the other hand I have 1Gbit symmetric and since I have a public IP address I could even allow direct connection if Resonite allowed it in some practical way. I'd be happy to share the fast, low latency link with as many people as possible, just to give them better experience. Currently that's only possible when I host the session.
Just to keep this issue updated, there currently exists an in-testing mod that allows clients and host to distribute encrypted assets over bittorrent.
https://codeberg.org/Orion_Moonclaw/Magnetite/
