jitsi-meet-helm
jitsi-meet-helm copied to clipboard
taktakpeops
[TESTING][GCP][GKE] Chart on GCP
Following a chat on the issue https://github.com/jitsi/docker-jitsi-meet/issues/565 for Jitsi Meet in K8S using this chart, I am moving the discussion specifically related to GKE here.
@ChrisTomAlx - could you share your findings here?
I will also provide you some support.
Hey @taktakpeops This looks fine to me except for the jibri part, because each jibri pod only supports one recording at a time and the cpu / ram optimal requirements for jibri are much more than other pods (4vcpu 8GB Ram is what I have read on the community forums). Also jibri requires some node level changes. Refer to this pr.
I was thinking we should maybe continue this discussion on your jitsi issue because the jitsi team is super helpful and they could help if we run into problems.
I would love for them to see the architecture you have created here, looks impressive..
One thing to add is that this architecture works only if you have multiple nodes in your cluster and each JVB instance has its own node because JVB will bind the public IP of the node. So some affinity rules are needed for the deployment of multiple JVB
Sorry I have been AWOL for a bit. So to do node level changes for proper jibri functioning (atleast in GKE), I am planning to create a daemonset which will run on all nodes before they start up any other pods.
https://stackoverflow.com/questions/57005971/how-to-use-customise-ubuntu-image-for-node-when-gke-cluster-create
Also I am creating the jibri yaml file with container post-start and pre-stop scripts to correctly change the /home/jibri/.asoundrc file in each of the containers. I will probably use hostpath to read and write the same file within all the containers on a node so they all know which sound cards are available and which aren't.
As I mentioned earlier the Jibri in the architecture can possibly be a group like JVB is correct?
Yes, indeed it can. Even if I think that it’s strongly tied to Jicofo + Prosody and I would see all of it in one pod.
About the multiple JVB, it used some pod anti-affinity rules to ensure the dispatch of the pods across nodes. Since each JVB deployment deploys every instance to its own pod with a port unique to the deployment it works with many Jitsi servers.
About Jibri, one issue remain: not all cluster can access the sound device of the VM.
I am starting to explore how to use a dummy sound driver that can be used as a loop back for recording sound from a pod without needing the physical device. I will most likely develop a K8S plugin device for that.
Let me know if you have more questions and feel free to push a PR with your scripts for GKE if you are using the chart :)
Sure I will start looking into this from tomorrow again. Will raise a PR if there are any changes required.. Regarding Jibri, a dummy sound device would be amazing. Jibri uses ALSA. If there was a way to add that to the container directly then most the jibri problems would be solved, but I am not entirely sure that is possible.
Hey @taktakpeops I noticed that you are using
- name: DOCKER_HOST_ADDRESS
valueFrom:
fieldRef:
fieldPath: status.hostIP
status.hostIP is only available in kubernetes version 1.17.0 and above, but it seems like GCP does not have a stable release for 1.17.0 yet, There is a rapid release available but google specifically mentions not to use it in production.. Does aws support 1.17.0 already ?
Hi @ChrisTomAlx,
Looking into the documentation here, it's available since v1.7. Checking here, it seems that GCP is currently supporting 1.15.9, 1.16.5, 1.17 and higher for GKE.
In AWS, I am using the latest version supported by EKS which is v1.16.8 (you can find all supported versions of K8S for EKS here: https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html).
Have you tried deploying and noticed that the IP wasn't exported in the environment?
Hi @taktakpeops Oh.. You are absolutely right. GKE does support 1.17 but it is not on a stable channel. But it does not really matter now that I know hostIP was actually implemented in 1.7.0+ and not 1.17.0+
Thanks for your quick response. You just saved me a whole lot of cluster reconfiguration
Hey @taktakpeops ..
I have a couple of questions. If time permits please do get back to me.
-
The status.hostIP uses the nodes IP address correct? But this IP address must be accessible from outside through an external IP. So my question is did you have to manually set up the nodes external IP ?
-
Each Prosody / Jicofo combo uses different xmpp domains right?
- name: XMPP_DOMAIN value: jitsi.meet - name: XMPP_AUTH_DOMAIN value: auth.jitsi.meet - name: XMPP_INTERNAL_MUC_DOMAIN value: internal-muc.jitsi.meet - name: XMPP_MUC_DOMAIN value: muc.jitsi.meet - name: XMPP_GUEST_DOMAIN value: guest.jitsi.meetFor example all the domains above should change based on the prosody / jicofo combo dynamically right? How did you manage to do that ?
-
I see some config files in your repo. Are those just to show that these are the configs within the containers? Or is it somehow being used in the helm chart ?
Hello,
For question 1: yes, you need to be able to remotely access your nodes which means that they need a public IP. In AWS, I did so by enabling the SSH access in my node group. The public IP are being detected using STUN servers.
Question 2, yes, you are right but for that I am writing the logic in helm which will generate the domains for each Jicofo / prosody couple (so work in progress).
Question 3, the config are there for testing. I want to start using octo and enable some modules in posody.
Awesome. Thanks for the quick response. There seems to be some issue on my end when using NodePort. The JVB is not accessible. Tried with LoadBalancer and it works.
kubectl describe pod {jvb pod} gives me :-
Name: jitsi-meet-jvb-0-7c5cdc6f54-slfn5
Namespace: default
Priority: 0
Node: gke-neumeet-pool-1-4768d01b-xd50/10.128.0.26
Start Time: Fri, 22 May 2020 17:15:34 +0530
Labels: app.kubernetes.io/instance=neumeet
app.kubernetes.io/name=jitsi-meet-jvb-0
pod-template-hash=7c5cdc6f54
Annotations: kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container jitsi-meet-jvb
Status: Running
IP: 10.56.4.199
IPs: <none>
Controlled By: ReplicaSet/jitsi-meet-jvb-0-7c5cdc6f54
Containers:
jitsi-meet-jvb:
Container ID: docker://7819c812939676fe72a614256ec57ce8f83a11ab0aa71c2121f9501878a90daf
Image: asia.gcr.io/webrtcvideo-274408/neutrinos/jvb:v20
Image ID: docker-pullable://asia.gcr.io/webrtcvideo-274408/neutrinos/jvb@sha256:a2e405eb74e33accc0e62ec1a798d918874b505fe58cd4f0e77bc0ac719fff43
Port: 30300/UDP
Host Port: 0/UDP
State: Running
Started: Fri, 22 May 2020 17:15:37 +0530
Ready: True
Restart Count: 0
Requests:
cpu: 100m
Environment:
XMPP_SERVER: jitsi-meet-prosody
PROSODY_INSTANCE: 0
JICOFO_AUTH_USER: focus
JICOFO_AUTH_PASSWORD: <set to the key 'JICOFO_AUTH_PASSWORD' in secret 'jitsi-meet-jicofo-config'> Optional: false
JVB_AUTH_USER: jvb
JVB_AUTH_PASSWORD: <set to the key 'JVB_AUTH_PASSWORD' in secret 'jitsi-meet-jvb-config'> Optional: false
JICOFO_COMPONENT_SECRET: <set to the key 'JICOFO_COMPONENT_SECRET' in secret 'jitsi-meet-jicofo-config'> Optional: false
JVB_PORT: 30300
JVB_STUN_SERVERS: stun.l.google.com:19302,stun1.l.google.com:19302,stun2.l.google.com:19302
JVB_TCP_HARVESTER_DISABLED: true
DOCKER_HOST_ADDRESS: (v1:status.hostIP)
JVB_OPTS: --apis=xmpp,rest
ENABLE_STATISTICS: true
XMPP_DOMAIN: testmeet.neutrinos.co
XMPP_AUTH_DOMAIN: auth.jitsi-meet-prosody.default.svc
XMPP_INTERNAL_MUC_DOMAIN: internal-muc.jitsi-meet-prosody.default.svc
XMPP_MUC_DOMAIN: muc.testmeet.neutrinos.co
XMPP_GUEST_DOMAIN: guest.jitsi-meet-prosody.default.svc
JVB_BREWERY_MUC: jvbbrewery
TZ: Europe/Amsterdam
Mounts:
/defaults from config (rw)
/var/run/docker.sock from dockersock (rw)
/var/run/secrets/kubernetes.io/serviceaccount from neumeet-jitsi-meet-token-tclkm (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
dockersock:
Type: HostPath (bare host directory volume)
Path: /var/run/docker.sock
HostPathType:
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: jitsi-meet-jvb-config-cm
Optional: false
neumeet-jitsi-meet-token-tclkm:
Type: Secret (a volume populated by a Secret)
SecretName: neumeet-jitsi-meet-token-tclkm
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 30m default-scheduler Successfully assigned default/jitsi-meet-jvb-0-7c5cdc6f54-slfn5 to gke-neumeet-pool-1-4768d01b-xd50
Normal Pulling 30m kubelet, gke-neumeet-pool-1-4768d01b-xd50 Pulling image "asia.gcr.io/webrtcvideo-274408/neutrinos/jvb:v20"
Normal Pulled 30m kubelet, gke-neumeet-pool-1-4768d01b-xd50 Successfully pulled image "asia.gcr.io/webrtcvideo-274408/neutrinos/jvb:v20"
Normal Created 30m kubelet, gke-neumeet-pool-1-4768d01b-xd50 Created container jitsi-meet-jvb
Normal Started 30m kubelet, gke-neumeet-pool-1-4768d01b-xd50 Started container jitsi-meet-jvb
Anyhow just dropped this here in case you find something off. I will look into stun servers I think that must be why I am having issues. Because I am able to ping the external ip of the node from an external network but not with UDP which is strange. So from a first look stun servers should solve this issue if I am not wrong.
@ChrisTomAlx, for the stun + public IP detection, you should see that in the logs of JVB.
Did you make sure that the security group for your nodes accept traffic on port 30300?
@taktakpeops Yea will look through the logs for some hints Its not a private cluster so I assumed nodes will be accepting traffic without problems.. let me recheck that. Thanks for the suggestions!!
@ChrisTomAlx : it’s not sure that the nodes will have the port open. In EKS, for example, the public network for the nodes has its own security group which has to be modified accordingly after deploying Jitsi to allow UDP traffic on the node.
Hey @taktakpeops you were right. There was a firewall blocking UDP traffic, trying to get around it. In the mean time I see someone else is also trying to replace ALSA dependency from Jibri with pulseAudio. Look at this comment.. Here is his github repo. I am not sure if it works, but since you said you were looking at removing node level changes from jibri, I thought you would be interested..
Hey @ChrisTomAlx - thanks for the suggestion. After playing around with Alsa (used currently in Jitsi) - I ended up realizing that it requires a modification of the node OS, therfore specialized nodes for running Jibri.
I saw quite a few other people also modifying the config of Jibri for using PulseAudio. After going through issues / pull-requests and the forum, I found out that PulseAudio was originally used in Jibri. The Jitsi team dropped it because of stability issues, while recording calls with a large amount of participants they had drops / packet losts. They replaced it by Alsa.
I want to keep the Helm chart with the official containers.
For now, I am looking into how the AMIs for EKS are configured and how to enable some specific kernel modules as the one missing currently: soundcore.
Once soundcore is enabled, we can run the node with an access to a Alsa dummy soundcard (ideally supporting X recordings at a time).
The custom AMI will be then used for all the nodes. If X recordings at a time aren't possible, we can still run one Jibri per nodes.
I thought also about another (weird and terrible) thing: running Jibri inside a VM running inside a container running Qemu 🤣
I thought also about another (weird and terrible) thing: running Jibri inside a VM running inside a container running Qemu 🤣
This is intriguing, but I am even scared to ask about it after reading up a bit on Qemu. Although it could work right? It will be one heavy pod but I think anything is better than making node level changes
I saw quite a few other people also modifying the config of Jibri for using PulseAudio. After going through issues / pull-requests and the forum, I found out that PulseAudio was originally used in Jibri. The Jitsi team dropped it because of stability issues, while recording calls with a large amount of participants they had drops / packet losts. They replaced it by Alsa.
Nice catch, would have spend a lot of time going down that rabbit hole otherwise.
I want to keep the Helm chart with the official containers.
Agree. I would prefer the official ones as well. Then we can simply change container images and stay up to date with new features..
For now, I am looking into how the AMIs for EKS are configured and how to enable some specific kernel modules as the one missing currently: soundcore.
For GCP I could change the node images to ubuntu then ssh into each of the VMs and change them manually as suggested by the Jitsi Team and this pr. Although this won't work in the case of node auto scaling. Hence my plan is to create a daemonset that run on all nodes and adjusts them to run jibri. Just a side note remember one jibri pod can only do one recording at a time. This video gave me a lot of insight into jibri. Leaving it here in case you want to watch it as well.
Also thanks a ton for the firewall tip.. Got JVB to run as expected.. Testing scaling next
This is intriguing, but I am even scared to ask about it after reading up a bit on Qemu. Although it could work right? It will be one heavy pod but I think anything is better than making node level changes
Yes - it could work but it would also be a hell to debug - it's a high level of inception :D
Nice catch, would have spend a lot of time going down that rabbit hole otherwise.
Agreed - it's pretty tricky to gather info about jitsi as it's all over the place
For GCP I could change the node images to ubuntu then ssh into each of the VMs and change them manually as suggested by the Jitsi Team and this pr. Although this won't work in the case of node auto scaling. Hence my plan is to create a daemonset that run on all nodes and adjusts them to run jibri. Just a side note remember one jibri pod can only do one recording at a time. This video gave me a lot of insight into jibri. Leaving it here in case you want to watch it as wel
For GCP, we might have the same issue as with AWS which is that the VM don't have a sound device. I am looking further into dummy-snd but it remains a kernel module and not included in Ubuntu on AWS EC2.
Will keep you posted once I find a workaround for that
I just followed the below steps as shown in the pr for each node on GCP and I was able to get jibri to run.
For each node in the cluster do the following:
Install generic kernel image
sudo apt-get update
sudo apt-get install linux-image-generic
Change grub to start node VM with generic kernel
# GKE Node
sudo vim /etc/default/grub.d/50-cloudimg-settings.cfg
# Other Setup
sudo vim /etc/default/grub
Replace #GRUB_DEFAULT=0 with
# Make the latest generic kernel the default
release=$(linux-version list | grep -e '-generic$' | sort -V | tail -n1)
GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux $release"
sudo update-grub
Reboot VM
sudo reboot
Setup virtual sound device in the node
install the module
sudo apt update && sudo apt install linux-image-extra-virtual
configure 5 capture/playback interfaces
sudo su
echo "options snd-aloop enable=1,1,1,1,1 index=0,1,2,3,4" > /etc/modprobe.d/alsa-loopback.conf
setup autoload the module
echo "snd-aloop" >> /etc/modules
Reboot
sudo reboot
check that the module is loaded
lsmod | grep snd_aloop
Hey @ChrisTomAlx, thank you for sharing.
I don’t see in your steps anything related to setting up the snd-dummy component, is it present in your nodes?
The Setup virtual sound device in the node section handles that I assume.
You can also refer this. The jitsi official documentation mentions how to set it up on AWS. Although I guess they meant it for VM's. In GKE the the kubernetes nodes are also VM's. I am not sure if its the same in AWS
For GCP if node auto scaling and node auto repair are on these changes won't stick. So I had to turn these off as well.
Here is my jibri yaml file. You might want to change it according to your needs slightly.. This is a one time only download link.
If you run « aplay -L » in your terminal, what does it print?
Also, master and nodes are VM. Inside of the node VM, kubelet must be running to subscribe to the master. Then, pod scheduling happens in this VM.
If you run « aplay -L » in your terminal, what does it print?
It just says -bash: aplay: command not found. But I do have a jibri pod running and working on this node. And all I did was follow the steps above and made sure the node image was ubuntu instead of Google's own node image
Also, master and nodes are VM. Inside of the node VM, kubelet must be running to subscribe to the master. Then, pod scheduling happens in this VM.
Yes, makes sense, similar to GKE.
Ah oki, i installed alsa on the VM to test my OS setup.
Got it to work on AWS. Now, for Jibri, how do you mount the device for the pods?
For the EC2 instance, I had to use a different GRUB_DEFAULT. Instead of the menu title, I used gnulinux-advanced-6156ec80-9446-4eb1-95e0-9ae6b7a46187>gnulinux-4.15.0-101-generic-advanced-6156ec80-9446-4eb1-95e0-9ae6b7a46187.
Could you share the logs from Jibri to see how alsa gets init and so on?
how do you mount the device for the pods?
Look at my jibri yaml file. Also /home/jibri/.asoundrc contains the loopback device being used. Refer the Set interface in file /home/jibri/.asoundrc inside a docker container section here
Could you share the logs from Jibri to see how alsa gets init and so on?
Sorry. I don't think its safe sharing, since I have jibri currently up on a UAT environment, so my hands are tied.
Jisti has mentioned how to change the GRUB file in the same link :-
# open the grub file in editor
nano /etc/default/grub
# Modify the value of GRUB_DEFAULT from "0" to "1>2"
# Save and exit from file
# Update grub
update-grub
# Reboot the machine
reboot now