Investigate Shared Resource Locking and Stuttering with Multiple Unity Clients

[galdalali]

We have a multiplayer architecture with a server and multiple clients (Unity exe application) all running on the same Windows machine. Each client implementing its own Cesium plugin instance. Each client is its own process with its own heap and Cesium instance.

We have observed severe stuttering and slowdowns in 3D tile loading whenever we run multiple client instances simultaneously (starting from the second client).

This issue appears as soon as another client is instantiated on the same machine, even if tile-loading parameters are tuned to extremely minimal values:

preloadAncestors = false; preloadSiblings = false; forbidHoles = true; maximumCachedBytes = 0; maximumSimultaneousTileLoads = 1; loadingDescendantLimit = 1; enableFogCulling = false; enableFrustumCulling = true; enforceCulledScreenSpaceError = false; generateSmoothNormals = false; createPhysicsMeshes = false;

When using a single client, tile streaming is smooth and stable, regardless of these parameter settings. However, as soon as a second client is launched, stuttering appears immediately, no matter the parameter tuning or tile source.

Observed Issues & Debugging Steps Taken:

• Even with maximumCachedBytes=0, a Cache.sqlite file is created.
• Disabled optional features like preloadAncestors, preloadSiblings, generateSmoothNormals, createPhysicsMeshes, etc.
• Tested against different data sources:
  • Our custom tilesets.
  • Google Photorealistic 3D tiles.
• Tested on different machines.
• Tested using Cesium for Unity version 1.16.0.

Key Findings:

• Stuttering only occurs when multiple clients are running simultaneously.
• Even extremely conservative tile-loader settings do not resolve the issue.
• The Cache.sqlite file is still created even if maximumCachedBytes=0 - Suggests that Cesium’s SQL cache might be shared or locking across clients (can be something else as well but this is our best guess), despite each client using its own Cesium plugin instance.
• Performance bottlenecks have been ruled out, the issue appears to be a synchronization or locking issue in Cesium's caching or tile loader internals making Unity main thread to stutter.

We’d appreciate your thoughts on this phenomenon and what might cause the stuttering. It’s definitely not a straightforward performance issue, it looks more like shared resource contention at the Cesium native level.

[azrogers]

@galdalali Thanks for the report! To clarify - you mention testing it on multiple machines. Does stuttering still occur when using multiple machines? I would expect this to rule out synchronization and locking issues.

[galdalali]

@azrogers Thanks for the follow-up. I haven’t tested on multiple machines yet, that would definitely help isolate the issue. That was going to be my next step as well. Given that the stuttering appears as soon as a second client is instantiated on the same machine, I’m 99% sure this is a shared locking or resource contention issue inside Cesium’s native caching or tile loader.

I’ll test this on separate machines and let you know as soon as I have results.

[galdalali]

@azrogers I can confirm that we see no stuttering at all when running the clients on different machines. The issue is clearly tied to resource contention or a shared lock when multiple clients run on the same machine.

[kring]

@galdalali can you try making a separate copy on disk of your application for each instance that you run? It sounds silly, but that may help avoid SQLite-related locking issues. If that works, it may be a useful workaround for you, and it also helps to narrow down where the problem is occurring.

[galdalali]

@kring I tried running a separate copy of the Unity build as suggested, but the issue still persists.

I also noticed that while Cache.sqlite is being created, it's only created once. I was expecting to see a separate Cache.sqlite file per client, but that doesn’t seem to be happening.