MirrorNetworking
SyncLists are limited in total data size to a single message packet in their underlying transport.
Describe the bug When a new client connects to an object with a sync list in it, the connection will fail if the current contents of the sync list cannot be delivered in a single message of the underlying transport. (e.g. > 250K or so for Kcp, about half that for Telepathy.). It fails with an error on the host/server that a message size is too large. This will also prevent scene network objects from being re-enabled and functioning at all.
250K is fairly small in some scenarios (number of "inventory" items lying about a mid-sized RPG world, for example), and it seems weird to require that the initial sync be atomic like that.
[IMPORTANT] How can we reproduce the issue, step by step: Please tell us how to reproduce your issue, STEP BY STEP, with one of our built in examples.
- Create a new host/client model using the HostUI, or use any client/host example. (assuming Kcp transport).
- Define a structure ("my_struct") with a few hundred bytes or so of data in it (say 25 integers or a couple of UUID strings).
- Create a gameobject with a Network Identity and a new component on it. Put it in the scene.
- In the new component, create a SyncList<my_struct>.
- On the server, create 5,000 of those objects, add them to the sync list.
- Attempt to connect with a second client. Note that the gameObject will disable at launch and never be re-enabled.
- Meanwhile, back on the host, you'll get a message about not being able to deliver a message > 250K or so.
Any bug that can be reproduced, can be fixed. If we can't reproduce it, we can't fix.
Expected behavior I'd expect this to sync, even if it takes a few messages to do so. Note that this scenario works:
- Wait to create the 5000 objects on the server until AFTER the client has connected. As long as the objects come in one or a few at a time, Mirror is more than capable of dealing with SyncLists that large, even quite performantly. It's just the initial everyting-at-connection-time sync that's failing.
Screenshots If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information): OS: MacOS Server, Windows client for sure, but I think any combination. Build Target: Mac, Windows Unity Version: 2021.3.4f1 Mirror Version: Installed from Package Manager/Asset Store and up to date as of now: 66.0.9
Additional context As a workaround, the Kcp transport added support for unlimited-size messages in Nov 2021 (and it's in the vis2k/kcp2k repository), but that never made it into Mirror. Pulling the newer version of the Kcp transport into Mirror would unblock this for Kcp only, but I think it should still be resolved.
250K or so for Kcp, about half that for Telepathy.
Telepathy's max message size can be higher, there should be an option in the inspector
Yeah, I'm doing that for now as a workaround, but the "reasonable size" for a collection data structure and the "reasonable size" for network packets are probably a 2-3 orders of magnitude different, and it doesn't really make sense for the former to be limited by the latter.
Mirror uses MaxMessageSize to fit everything. the larger it is, the more fits. we recommend keeping it reasonable (64 KB is quite a lot).
however as mentioned, you can increase transport size quite a lot. even 2 GB etc. for telepathy.
let me know if that makes sense. otherwise feel free to reopen :)
So I feel like maybe I'm not expressing myself well here, because I think we're in agreement but reaching opposite conclusions.
My fundamental argument here is that the "cost" (expressed here as message size) of networked collections should be relative to the size of the elements of those collections and the number of them that change in a given time period, not the entire size of the collection.
As an example, if I have hundreds of thousands of X in my world, each of which is a few hundred bytes, but players only change/interact with a few dozen of them every minute, the actual data that needs to be exchanged during gameplay is on the order of a few KB at most per message. My assertion is that we should be able to build our MaxMessageSize around that usage. That's the scenario in which "64 KB is quite a lot."
But there's one outlier: When a new player joins the game, the entire collection needs to be delivered to that player. This occurs only once, when the initial synchronization of the data occurs, but in this single case, the entire collection is currently constrained in MIrror to fit inside a single message packet. That makes sense for a SyncVar, but not for a SyncList or SyncDictionary.
There's no way for the dev to work around this; in order to use a large (or really even medium) collection (even if no large modifications are ever made to it-- which describes most games), I need to choose a transport and message size large enough to contain the collection as a whole, not the actual usage of it. I can't choose to deliver it in "chunks" over a few frames, move the data separately in this one case and fill it in manually, pre-load it from local data, or any other optimization. To me, the list is opaque and monolithic, and this scenario both prevents me from using most transports altogether or specifying absurdly large MaxMessageSizes on the others. And "medium" or "large" here describes things that are very, very, very small in terms of the data structures used in games: 64KB is not "quite a lot" for collections of terrains, monsters, models, items, buildings, spawns, etc. in an open-world game.
The one thing I can do is break the collection into a lot of smaller collections and then reconstruct them on my side, but that adds a great deal of complexity in order to work around what I consider a design flaw in the network layer.
Very specifically, what I'm suggesting here:
- There's nothing wrong with Mirror constraining updates to a collection fitting in MaxMessageSize/delivery. That makes perfect sense.
However:
- The initial synchronization of data (where the entire collection needs to be delivered to a new connection for the first (and only) time) should be able to complete in multiple packets, over as long a period as is necessary. There's no chance for "partial data" use here, since Mirror won't enable the NetworkBehavior object on the client end until it's done, anyway. And it won't affect performance of existing solutions, because right now if we're exceeding the MMS, Mirror just fails to ever activate the client object.
As I write this, I'm sort of questioning my algorithms, because it seems weird that I'd be the only person requesting this. It seems like a limitation that would be bumped into by effectively everybody writing games having shared worlds with lots of player-manipulable "whatevers" in them. I'd expect this to be an extremely common request, and I'm not entirely sure why it isn't, or if everybody's just living with it by using a transport with a massive message size cap.
It's just a weird behaviour: normal (non-network) collections don't usually change their performance based on how many elements are in them, just how many you read/write. The total collection size matters only really for memory use, and it would be very bizarre if, for example, Dictionary<X,Y> or List<W> had a limit that the entire collection had to fit in a single memory page or it couldn't be constructed, which is sort of what's happening here.
Effectively, by requiring that SyncList/SyncDictionary fit in a single packet, you've conceptually turned them into simple SyncVars. Aside from the performance hit, they would work exactly the same if the entire collection were re-delivered in every packet, and in fact that's how they DO work at initial creation.
So...I still think this should get addressed, and that Mirror 2 should design it in from the start.
But I'm not going to be belligerent about it. I am reopening this with my hopefully-more-compelling argument here, but if you close it again, I'll leave it be.
(Rather, I'd like you to reopen it. Since this was closed by a collaborator, I am not allowed to).
There's no way for the dev to work around this
Of course there is ways to work around this. You dont have to use the built-in collections, You can make a custom collection that syncs things the way you need to.
It sounds like SyncList/SyncDictionary is the wrong tool for what you are trying to do. You'd probably be better using custom network message if you want a greater control over how things are sent.