Sathvik-Rao
[Feature Request] Server rewrite with session persistence
The java server is using 100MB per WS connection and grows to 1,5 GB after some usage. I don't know much java but I could make a GO implementation as an example which coule make implementing features easier in the future. Also the CPU usage during login is quite hígh. For a possibly multiuser server it should be made more efficient.
Would you think about a different language for the server? Do you have any preference for the language?
Hi @archef2000,
Could you help me reproduce your scenario?
I’m running Docker with the ClipCascade server (tag: 0.1.0) on Windows. I have multiple devices connected, and the RAM usage is around 188MB.
Could you clarify the following:
- Which Docker tag are you using (0.1.0 or latest)?
- On which OS is Docker running?
- Does your RAM usage increase exponentially?
- How long has your container been running?
- Do you see the RAM increase instantly after a new client device connects?
- Can you share the server log containing the line:
clipcascade-1 | 2024-10-07T14:14:24.289Z INFO 1 --- [ClipCascade] [MessageBroker-1] o.s.w.s.c.WebSocketMessageBrokerStats : WebSocketSession...at the time when the memory size reached 1.5GB? The log provides information about the number of devices connected, the number of messages sent, and it logs data every 30 minutes.
Thanks!
I am using: sathvikrao/clipcascade:0.1.0-multi-arch@sha256:7d1494d9d03142f6dd341d4c884d9cb93fda716e821094ddc2a1d69690d1088b also tested with sathvikrao/clipcascade:0.1.0
OS: Ubuntu Server 24.04 LTS The container boots up with 170MB and goes down to 140MB Loading the webinterface does nothing memory wise. As soon as I click Connect it goes up 80MB-100MB and stays even when disconnecting. I saw the 1.1 GB after experimenting a lot as I had problems with my phone so about 4 days, but I can quickly get to half a GB with my phone,tablet,windows client and two browser connections in just 5 min.
Also I get logged out everytime I recreate the container.
I have already recreated the container so nothing to share. Will try to get it to that point again.
Each WebSocket connection creates a thread since it is live broadcasting, which consumes memory. When disconnected, it should eventually release the memory, though not immediately, as garbage collection happens automatically. I am reproducing this and will update you. Also, could you let me know how much RAM you have physically and docker specific limit? A container restart will log out devices because the session keys are removed.
Have not limited the container yet. I have 32GB of ram available but I am currently only using 7GB with 60 containers.
Hi @archef2000, have you encountered the issue again? I haven't been able to exceed 350 MB, and it's been 3 days with 3 connections. I'm using the ContainerWatch extension in Docker to track the last 3 days of usage.
Currently using 950MB with the latest log statement:
INFO 1 --- [ClipCascade] [MessageBroker-2] o.s.w.s.c.WebSocketMessageBrokerStats : WebSocketSession[0 current WS(0)-HttpStream(0)-HttpPoll(0), 54 total, 0 closed abnormally (0 connect failure, 0 send limit, 44 transport error)], stompSubProtocol[processed CONNECT(54)-CONNECTED(54)-DISCONNECT(0)], stompBrokerRelay[null], inboundChannel[pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1323], outboundChannel[pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 556], sockJsScheduler[pool size = 6, active threads = 1, queued tasks = 0, completed tasks = 172]
Before restarting your container, did you happen to check how much RAM it was consuming?
Perhaps it is using a lot of RAM because you have plenty available, but it doesn't grow indefinitely. If you notice it growing indefinitely, please reopen this issue. Below is my container usage from the past three days.
Hi, @archef2000 I believe with the latest release, the container memory usage should be less. Because previously there was no heart beat so every 100 seconds it was creating a new connection. Now I added a heartbeat to the server so the connection starts alive and no high RAM usage.
I would like to add multiuser support and maybe even OIDC, but I am not very fluent in java and would prefer golang for speed, efficiency and many packages helpful for OIDC and persistence. Would you accept a PR for a server written in golang with multiuser support? How would you manage multiple users? First user is admin with a DB for all the users and maybe future history? A json file?
Thank you for your interest and suggestions regarding adding multiuser support and OIDC integration. I truly appreciate your enthusiasm for contributing to the project!
I wanted to share that the implementation for multiple users has been completed. The new setup includes a default admin user, and once logged in, the admin can update the username and password—eliminating the need for Docker environment variables. The admin can also create and manage multiple users, with each user having their own WebSocket connection.
The clipboard history feature is planned and will be implemented after the encryption update. This update focuses on enhancing security by ensuring that raw passwords never leave the client device, utilizing double hashing for added protection.
All users clipboard history (once implemented), will be securely stored in PostgreSQL, with each user linked to their own table of data for better organization and security.
These updates will be released as version 2.0.0, introducing significant changes that won’t be backward compatible with earlier versions.
While I appreciate your suggestion for a Golang-based server rewrite, the current redesign has already required significant development effort, particularly with the recent encryption enhancements. As such, migrating to Golang isn’t planned at this stage, but I remain open to future contributions and ideas for improving the project further.
Currently working on React for server front end and significant updates on all client devices change CSRF, Hashing, etc.
However, if you're interested, you can help implement any additional features, such as in the to-do list in polling discussion page or related enhancements, using the same codebase without requiring a full redesign.
Thank you
@Sathvik-Rao The version 3.0 still uses 544MB of RAM after running 2 days already. With no connections at all as I am getting signed out of it on windows and android. (Using the local database as I couldn't get the psotgres version to work. Sill need to try it again)
Are you running in P2P mode? For few users default database is fine. The postgresql is used for reliability when running for 1M users.