peertube-plugin-livechat
peertube-plugin-livechat copied to clipboard
JohnXLivingston
Requesting XMPP support not be removed or ejabberd used in place of Prosody
hey again friend,
in your 5.6.0 patch notes you specify that you are removing support for external XMPP servers and I'd like to toss in our experience with this.
On PoastTV I have on several occasions had to stream to audiences in excess of 600, sometimes over 1000 (in one case 2000 users) and Prosody stops responding to most activities around 350 users and completely fails to respond at all past 600.
Prosody's limitation is that it's single threaded and once you max out that CPU thread it has no room to breathe and just outright fails. Ejabberd is multi-threaded alternative that we planned to implement into PoastTV that will allow us the ability to scale to hundreds or thousands of concurrent users per stream.
While I think these numbers are quite low (suggesting perhaps some configuration or environment issue - I know of plenty of Prosody instances serving many thousands of users without a sweat) , I do think that at least having "advanced" route for people who want to connect with an external server for whatever reason would be beneficial.
I'm curious - what are the potential difficulties in preserving that option?
We have in production a Converse-based webchat (same upstream lib as this Peertube plugin) which uses Prosody and which sometimes has multiple thousands (but below 10k) users in a groupchat without issue.
We do have a few optimizations, the main one is to disable presence broadcasting (which can generate a lot of unnecessary traffic if you don't need it). We do this with mod_muc_defaults and we set presence_broadcast = {}.
GIven that (AFAIK) the PeerTube chat doesn't show chat occupants, you could probably use this optimization as well without issue.
Another thing (non-backend related) I would recommend for very large active chats is to set prune_messages_above for Converse.
This removes older chat messages from the local cache, making them ephemeral (unless you have message archiving enabled on the server, then they can be fetched again via infinite scrolling). This helps to avoid letting the chat history get too full and thereby slowing things down.
hey again friend,
in your 5.6.0 patch notes you specify that you are removing support for external XMPP servers and I'd like to toss in our experience with this.
On PoastTV I have on several occasions had to stream to audiences in excess of 600, sometimes over 1000 (in one case 2000 users) and Prosody stops responding to most activities around 350 users and completely fails to respond at all past 600.
Thanks for your feedback. I spoke with @Chocobozzz last week-end, and there is a lack of feedbacks for such streams. It is very complicated for us to find instances that have live streams with hundreds of viewers. It would be very appreciated if - for example - you would write a post on the Peertube forum https://framacolibri.org/c/peertube/38 to tell about your experience. How many viewers, if Peertube did well, if the chat did well (apparently not :( ). If you have some metrology (how many bandwitch used on the server, CPU usage, ...). And so on. So that we can know if it works or not, and improve the softwares :)
Prosody's limitation is that it's single threaded and once you max out that CPU thread it has no room to breathe and just outright fails. Ejabberd is multi-threaded alternative that we planned to implement into PoastTV that will allow us the ability to scale to hundreds or thousands of concurrent users per stream.
As said by @mwild1 and @jcbrand , Prosody should handle this properly. I think the problem is elsewhere. The way Prosody is served to the browser is very particular in this plugin. In order to minimise the server configuration, I made the following choice: Peertube acts as a reverse proxy, and I use the BOSH protocol (not websocket). To do that, I use this package: https://www.npmjs.com/package/express-http-proxy Here is the code: https://github.com/JohnXLivingston/peertube-plugin-livechat/blob/d24ef987a03bb781f95c482a74444ff5c7267b50/server/lib/routers/webchat.ts#L234-L239 And: https://github.com/JohnXLivingston/peertube-plugin-livechat/blob/d24ef987a03bb781f95c482a74444ff5c7267b50/server/lib/routers/webchat.ts#L338
Two problems:
- BOSH is less effective than websocket
- Peertube is single process
From the beginning I was concerned that this might cause performance problems. But I did not managed to set up a scalability test environment, and I was lacking feedback from Peertube instances admins. I was not aware of any chat with more than 200 users. You are the first to mention such problems (and many thanks for that!).
I see three potential solutions:
- try to use websocket instead of BOSH. I don't know if this is easy to achieve (I have to do some R&D)
- maybe try to run other NodeJS process to handle the reverse proxy (again, some R&D needed).
- for big Peertube's instance, with hundred of viewers, document how to bypass the Peertube reverse proxy mechanism (this mechanism is here for simplicity, but it can be bypassed with some nginx configuration tweaking) [1]
Thanks again for your feedback. I'll try to find a solution before cutting the «external XMPP» support.
[1] If you plan to have other big live streams, you can try to add this in your nginx configuration:
location /plugins/livechat/5.6.0/router/webchat/http-bind {
proxy_pass http://localhost:52800/http-bind;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_buffering off;
tcp_nodelay on;
}
Replacing 5.6.0 with the actual plugin version, and 52800 with the Prosody port as configured in the plugin settings.
If this solves the performance problems, I can try to document this (and remove the need for the version number in the path).
We have in production a Converse-based webchat (same upstream lib as this Peertube plugin) which uses Prosody and which sometimes has multiple thousands (but below 10k) users in a groupchat without issue.
Thanks for the feedback!
We do have a few optimizations, the main one is to disable presence broadcasting (which can generate a lot of unnecessary traffic if you don't need it). We do this with mod_muc_defaults and we set
presence_broadcast = {}.GIven that (AFAIK) the PeerTube chat doesn't show chat occupants, you could probably use this optimization as well without issue.
The chat show the occupants (but it is hidden by default). I take good note of your advice.
Another thing (non-backend related) I would recommend for very large active chats is to set prune_messages_above for Converse.
Ok, I will copy this in a new issue, and try this option (I plan to work on the plugin next month).
Thanks @jcbrand , your help is always appreciated (And ConverseJS is really a great software!)
Thanks @JohnXLivingston
The chat show the occupants (but it is hidden by default)
Ok, if you turn off presence broadcasting, then you won't see any occupants, so sounds like this isn't a (general) solution then.
on Friday we had another stream where about 400 people were participating in chat and it was unusable. I wonder if maybe adding a slow chat mode would be beneficial in this situation? I think it can be configured to rate limit chats in prosody's configuration
on Friday we had another stream where about 400 people were participating in chat and it was unusable. I wonder if maybe adding a slow chat mode would be beneficial in this situation? I think it can be configured to rate limit chats in prosody's configuration
Unusable because of too many people speaking, or were there connection issues? Yes, I can try to activate rate limits. Can you open a new issue for that please? I'll take a look in a couple weeks. Thanks for your feedback.
not a connection issue -- too much traffic caused the prosody process to lock up repeatedly making chat unusable. could not even load the applet.
for tracking, Issue #105 is the request. Thank you
not a connection issue -- too much traffic caused the prosody process to lock up repeatedly making chat unusable. could not even load the applet.
I think this is not a Prosody issue, as I said in this comment: https://github.com/JohnXLivingston/peertube-plugin-livechat/issues/95#issuecomment-1092864770
Have you tried the solution at the end of this comment? (adding a bypass in the nginx conguration).
I suspect that the usage of BOSH plays a role here. It's much slower and more complex than websocket connections.
I suspect that the usage of BOSH plays a role here. It's much slower and more complex than websocket connections.
Totally agree. I did not say it, but it is on my roadmap to allow websocket. I hope I can do it this month.
yeah so there's about 1400 people right now https://poast.tv/w/etozLs62Mcep2oGJL5yMEo
i cant even load the page when the chat plugin is loaded. if i disable it, it loads fine this is with the "fix" in comment: #95
Ok, thanks for the feedback. I'll try to release a version using websocket next week. I hope it will help.
Have you got some metrology on the server? It would be useful to see some graphs about CPU usage, network, etc. ; to see where is the bottleneck (to be sure we are not on a wrong track). If you have nginx error logs, it could also be helpful.
there's no error logs. the server just dies. I'll get you logs next Friday. it's a weekly stream on Friday at noon. I'll @ you on the fediverse but that is dual xeon sever with NVMe and 64GB ram on 40gbps link
Ok, thanks. I'll do my best to release a new version of the plugin before friday, so you can test it.
@animegrafmays , for the record I'm currently working to enable Websocket. This implies to rewrite the way I proxifies BOSH request too. My work is on this branch: https://github.com/JohnXLivingston/peertube-plugin-livechat/tree/prosody_proxy_optimization BOSH is rewritten and is ok. Websocket is written, but does not work for now (I don't understand why). I hope I can fix this by tomorrow, so you can test it friday.
I appreciate the effort. I was talking to someone who is familiar with or knows people who did something similar with prosody and websockets as well, he said the following in our Matrix chat:
https://prnt.sc/QigkRO4dFvI-
I am hopeful it will but I truly think that for something like this we will have to use ejabberd or something that we can scale, but I do not wish to because we want to use built in function you have added to this plugin for prosody
I don't totally exclude to use ejabberd. I saw that it comes with a Linux RUN Installer. That's something that is missing for Prosody, and I'm struggling to find a solution to automatically install Prosody from the plugin... With ejabberd, it would be very easy.
But if I switch to ejabberd, I have to rewrite the code from my Prosody plugins. This should not be very difficult, but I don't know if ejabberd has a plugin system, and if it permit to do what I need. I have to read some documentation to clarify my ideas.
Please note that switching to ejabberd will not magically resolve all performance issue. Ejabberd scalling will not be possible out of the box. And I will have the same proxy problem: I want to offer a zero configuration setup for peertube admins, I have to proxify the request through Peertube (which is single process too...). No worry, I plan to offer advanced configuration possibility, for high usage instances.
For the record, I think it would be possible to scale Prosody. In a very near feature, I plan to handle federation, and connection between instances... But, I think it would be possible to have multiple Prosody on the same server, so we can load balance users between multiple Prosody server (as Prosody is single threaded, it would help to have 4 or 5 difference Prosody on the same server).
In short, to summarize, I have several leads to explore. Unfortunately it will take some time, and it will require to proceed step by step, to be sure to spot the bottlenecks, and to find the right way to manage them - one by one.
thats fine, send me an email to [email protected] and I will provide you with credentials to the main peertube server so you can also see what I mean on Friday
I can't be there this friday, sorry.
I'll try to release the websocket stuff tomorrow (I'm in France, UTC+2 ; it is midnight right now, i'll start to work on it tomorrow morning). But not 100% sure to achieve it in time. I'll let you know.
Sorry @animegrafmays , I can't achieve this development this week. Websocket is more difficult to proxify that I meant. I have it (almost) work on my dev server, but in production condition it doesn't work. It seams that the problem comes from the 2-layer reverse proxy: nginx + Peertube. I'll try to do it next week.
take your time, I think we may reschedule this week's stream anyway due to ongoing issues outside of our control. i appreciate your efforts
I'll let you know when I have something working in production. Thanks.
@JohnXLivingston if you find Prosody installation from packages difficult and you need help finding an alternative solution, feel free to reach out ( [email protected] ) with what you need and we can probably point you in the right direction. Prosody has been installed in all kinds of environments, and although we strongly recommend OS packages to most users for simplicity, there are certainly other deployment options.
@JohnXLivingston if you find Prosody installation from packages difficult and you need help finding an alternative solution, feel free to reach out ( [email protected] ) with what you need and we can probably point you in the right direction. Prosody has been installed in all kinds of environments, and although we strongly recommend OS packages to most users for simplicity, there are certainly other deployment options.
Thanks @mwild1 . Here is a summary of my needs:
- My plugin should work without any server configuration, or server access (for basic usages, that does not mean that we can do some tweaking for advanced users)
- Prosody and all required libs (lua, ...) must be installed when this Peertube plugin is activated (that is done from the Peertube web interface)
- I have no root access, Peertube runs with a dedicated unprivileged user.
- I have a folder in which I can do what I want, with this user rights. But, the path of this folder is not predictable! Last time I checked, the Prosody Makefile need to have some absolute path for some libs, and I was stuck there.
- It must run on several architecture (amd64 and arm)
- It must run on any UNIX/like system, with minimal packages already installed. I can't install any additional package (so I probably won't have access to «make» on the server for example)
Once I have the Prosody executable, my plugin generates a config file. For now, it only listen on localhost, and on specific ports. And I proxify every request in the Peertube web server.
A friend told me about a packaging system called nix (not 100% sure this is the correct link). I have to check, but it could probably work.
I will work on this in a couple weeks. I'll contact prosody devs if I don't find an easy solution. Thanks.
PS: for now, Peertube instance admin have to install the Prosody package. But that is something we want to avoid, so that server access is not needed to activate this plugin.
@animegrafmays , maybe the bottleneck is the number of connections allowed by nginx. Please try to increase nginx worker connections: https://linuxhint.com/what-are-worker-connections-nginx/
(see also here: https://framacolibri.org/t/nginx-optimization-for-1k-2k-visitors-for-live-stream/ )
yes. thank you I have already had considered this last year when I deployed peertube
www-data@stream:/home/graf$ ulimit -n
200000
events {
worker_connections 200000;
use epoll;
multi_accept on;
}
Another thing I think might cause issues with very large rooms is fetching of occupant VCards in order to show avatars.
There isn't currently a config option for Converse to disable VCard fetching for groupchats, but you can disable the VCard plugin itself.
When calling converse.initialize, pass in:
blacklisted_plugins: ['converse-vcard']
@JohnXLivingston: I think this might be something you can do by default for this project since AFAIK VCards are not used or relevant.
This will definitely reduce unnecessary traffic and strain on the XMPP server.