CloudNetService
Permissions not working on high players
Stacktrace
No error on console.
We have been observing for several weeks now that the permissions are not functioning when there are approximately 300 players (+/- 20). These 300 players are not spread across multiple servers but are on one specific subserver (Citybuild). If we just restarted and there are 300 players, it still works. But if we wait a bit, it stops working again. However, if we wait until there are about 250 players, it works again. We are wondering if it is CloudNet or Paper 1.19.2 that is causing this problem...
Actions to reproduce
- One Minute After Restart (300 Players) (14:38)
- 10 Minutes After Restart (300 Players). (14:48 Permissions not working)
These are my permissions. But it's not just me who can't do anything - normal players can't go to plots, for example.
CloudNet version
[03.01 14:47:52.004] INFO: CloudNet Blizzard 4.0.0-RC5 4ec2a84d [03.01 14:47:52.004] INFO: Discord: https://discord.cloudnetservice.eu/ [03.01 14:47:52.004] INFO: [03.01 14:47:52.004] INFO: ClusterId: *** [03.01 14:47:52.004] INFO: NodeId: Node-1-Testserver [03.01 14:47:52.004] INFO: Head-NodeId: Node-2-Testserver [03.01 14:47:52.004] INFO: CPU usage: (P/S) 1.91/4.49/100% [03.01 14:47:52.004] INFO: Node services memory allocation (U/R/M): 11264/19456/16000 MB [03.01 14:47:52.004] INFO: Threads: 75 [03.01 14:47:52.004] INFO: Heap usage: 54/256MB [03.01 14:47:52.004] INFO: JVM: Private Build 17 (OpenJDK 64-Bit Server VM 17.0.5+8-Ubuntu-2ubuntu120.04) [03.01 14:47:52.005] INFO: Update Repo: CloudNetService/launchermeta, Update Branch: beta
Other
No response
Issue uniqueness
- [X] Yes, this issue is unique. There are no similar issues.
As soon as there are fewer than exactly 300 players, it works again.
Exactly at 300 players.
Yes, "tps" and other stuff are not working. But /server still working
I need to take a deeper look then, cause the same code that runs on the proxies is used on the servers to calculate the permissions...
We have also tried what happens if I have OP. Before, it did not work (with * and all other permissions) but with OP it works immediately.
We increased the slots from 300 to 800 to prevent the server from filling up. The permissions worked again immediately. We let the server run at full capacity with 500 slots and captured the slots back to 300. The permissions stopped working again immediately....
Every day at 04:30, the entire network (each subserver) is restarted. Nothing is typically restarted or reloaded during the day.