bbb-install icon indicating copy to clipboard operation
bbb-install copied to clipboard

Azure Vm stops working after bbb installation 3.0

Open mansoor128 opened this issue 7 months ago • 10 comments

sudo apt-get install -y language-pack-en
sudo update-locale LANG=en_US.UTF-8
    sudo systemctl set-environment LANG=en_US.UTF-8
    wget -qO- https://raw.githubusercontent.com/bigbluebutton/bbb-install/v3.0.x-release/bbb-install.sh | sudo bash -s -- -w -v jammy-300 -s bbb-001.mywebsite.com -e [email protected]

This is my installation script, I am installing on ubuntu 22 vm on azure, The isntallation is working fine, and server also runs fine. But the issue is if i restart the server, it stops responding. I have tested mutiple time, If I donot retart the VM, It keeps working flawlessly, but as sson as i reboot the server, the vm stop responding and i can even ssh into the vm, It says timeout.

mansoor128 avatar Jun 11 '25 22:06 mansoor128

but as sson as i reboot the server, the vm stop responding and i can even ssh into the vm, It says timeout.

Hmm ... this sounds above BigBlueButton. Whatever is preventing you from ssh back into your VM is preventing BigBlueButton from working.

If you can get the ssh working (perhaps assign the VM a public IP address with static allocation), there is a good chance BigBlueButton should work as well.

ffdixon avatar Jun 12 '25 10:06 ffdixon

Image The issue I am facing is that the azure vm agert stops wroking, and the VM went into not responding state, I donot have the azure support, The issue seem to be from azure cloud side, but does any one have faced it?

mansoor128 avatar Jun 12 '25 20:06 mansoor128

Did you check the firewall? Some VMs have KVM access available on Azure, did you check if you can see anything on KVM?

GhaziTriki avatar Jun 12 '25 20:06 GhaziTriki

There is something that bigbluebutton do to the VM, because When I reinstalled it on the new server, I got error at the ssl certificate stage.

Image I restarted mutiple times after it, and the VM was working fine. But when Reinstalled the Bigbluebutton again after fixing the ssl issue. the VM stops responding again.

The Issue is once the BBB is installed, It works perfectly, and there are no issues with the VM also. I have checked the azure agaent it is also wroking, fine, no issues in any syslog or other logs, It only occurs, if I restart the server, The server donot comes back and throws the error on Azure page, which I shared above. -- Machine agent status is not ready.

mansoor128 avatar Jun 12 '25 20:06 mansoor128

Can you please check the network settings before and after the install? The content of files inside /etc/netplan. The machine might be coming back but not showing the Azure network, the agent then is not connecting the hypervisor. This is the only clue I can think of right now.

GhaziTriki avatar Jun 12 '25 20:06 GhaziTriki

Thank you pointing out, It seem there is some issue after restarting with network, If I connect via serial to VM this the error [ 66.629839] cloud-init[744]: 2025-06-13 12:01:19,246 - azure.py[WARNING]: Polling IMDS failed attempt 2 with exception: UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /metadata/instance?api-version=2021-08-01&extended=true (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x73a119c047f0>, 'Connection to 169.254.169.254 timed out. (connect timeout=30.0)'))")

Can you suggest how do I fix it, before restarting the machine, so It donot get into this state.

Before restarting the machine i did a curl -H "Metadata:true" "http://169.254.169.254/metadata/instance?api-version=2021-02-01"

and it was working fine.

mansoor128 avatar Jun 13 '25 12:06 mansoor128

Thankyou all for offering help. the issue is with the UFW , I have disabled it and it starts working, contrary to bbb documentations, where it is only mentioned to open ports, the complete firewall should be disable, I havent debug, what rules does the bbb installation changes.

mansoor128 avatar Jun 13 '25 18:06 mansoor128

I installed BBB3.1 on Azure, and enabled UFW, but did not encounter your problem. But I disabled Azure's automatic update, otherwise it would automatically replace some of BBB's software, rendering BBB unusable.

`Welcome to Ubuntu 22.04.5 LTS (GNU/Linux 6.8.0-1029-azure x86_64)

  • Documentation: https://help.ubuntu.com
  • Management: https://landscape.canonical.com
  • Support: https://ubuntu.com/pro

System information as of Thu Jun 19 05:49:14 AM UTC 2025

System load: 0.0 Processes: 257 Usage of /: 16.6% of 61.84GB Users logged in: 0 Memory usage: 32% IPv4 address for eth0: 10.18.1.8 Swap usage: 0%

  • Strictly confined Kubernetes makes edge and IoT secure. Learn how MicroK8s just raised the bar for easy, resilient and secure K8s cluster deployment.

    https://ubuntu.com/engage/secure-kubernetes-at-the-edge

Expanded Security Maintenance for Applications is not enabled.

13 updates can be applied immediately. 13 of these updates are standard security updates. To see these additional updates run: apt list --upgradable

25 additional security updates can be applied with ESM Apps. Learn more about enabling ESM Apps service at https://ubuntu.com/esm

New release '24.04.2 LTS' available. Run 'do-release-upgrade' to upgrade to it.

root@azmeet:~# ufw status Status: active

To Action From


OpenSSH ALLOW Anywhere
Nginx Full ALLOW Anywhere
16384:32768/udp ALLOW Anywhere
3478 ALLOW Anywhere
OpenSSH (v6) ALLOW Anywhere (v6)
Nginx Full (v6) ALLOW Anywhere (v6)
16384:32768/udp (v6) ALLOW Anywhere (v6)
3478 (v6) ALLOW Anywhere (v6)

root@azmeet:~# `

michaelchen001 avatar Jun 19 '25 06:06 michaelchen001

This issue is indeed triggered by UFW, when installed by BBB bbb-install.sh script, and by no means limited to Azure, I'm facing it on my very own Proxmox Virtual Environment 8.4.1-powered server behind Mikrotik router. Problem persists after reinstallation.

When you've just installed BigBlueButton (say, 3.0.12), everything works fine, both VPS and BigBlueButton. UFW settings looks fine as well: sudo ufw status verbose Status: active Logging: on (low) Default: deny (incoming), allow (outgoing), deny (routed) New profiles: skip

To Action From


22/tcp (OpenSSH) ALLOW IN Anywhere
80,443/tcp (Nginx Full) ALLOW IN Anywhere
16384:32768/udp ALLOW IN Anywhere
3478 ALLOW IN Anywhere
22/tcp (OpenSSH (v6)) ALLOW IN Anywhere (v6)
80,443/tcp (Nginx Full (v6)) ALLOW IN Anywhere (v6)
16384:32768/udp (v6) ALLOW IN Anywhere (v6)
3478 (v6) ALLOW IN Anywhere (v6)

sudo systemctl is-enabled ufw enabled sudo systemctl status ufw ● ufw.service - Uncomplicated firewall Loaded: loaded (/lib/systemd/system/ufw.service; enabled; vendor preset: enabled) Active: active (exited) since Tue 2025-07-22 18:38:39 UTC; 20h ago Docs: man:ufw(8) Main PID: 582 (code=exited, status=0/SUCCESS) CPU: 3ms

Jul 22 18:38:39 r710-bbb30-2 systemd[1]: Starting Uncomplicated firewall... Jul 22 18:38:39 r710-bbb30-2 systemd[1]: Finished Uncomplicated firewall.

However, when you reboot VPS, it'll be stripped off any connectivity to Internet and to local network, you can't connect via SSH. UFW is broken: sudo systemctl status ufw [sudo] password for uadmin: × ufw.service - Uncomplicated firewall Loaded: loaded (/lib/systemd/system/ufw.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Wed 2025-07-23 15:09:04 UTC; 1h 38min ago Docs: man:ufw(8) Process: 600 ExecStart=/lib/ufw/ufw-init start quiet (code=exited, status=1/FAILURE) Main PID: 600 (code=exited, status=1/FAILURE) CPU: 106ms

Jul 23 15:09:03 r710-bbb30-2 ufw-init[646]: Try iptables-restore -h' or 'iptables-restore --help' for more information. Jul 23 15:09:03 r710-bbb30-2 ufw-init[654]: iptables-restore v1.8.7 (nf_tables): Chain 'ufw-skip-to-policy-input' does not exist Jul 23 15:09:03 r710-bbb30-2 ufw-init[654]: Error occurred at line: 30 Jul 23 15:09:03 r710-bbb30-2 ufw-init[654]: Try iptables-restore -h' or 'iptables-restore --help' for more information. Jul 23 15:09:03 r710-bbb30-2 ufw-init[677]: iptables-restore: line 5 failed Jul 23 15:09:04 r710-bbb30-2 ufw-init[725]: Problem running '/etc/ufw/before.rules' Jul 23 15:09:04 r710-bbb30-2 ufw-init[725]: Problem running '/etc/ufw/after.rules' Jul 23 15:09:04 r710-bbb30-2 systemd[1]: ufw.service: Main process exited, code=exited, status=1/FAILURE Jul 23 15:09:04 r710-bbb30-2 systemd[1]: ufw.service: Failed with result 'exit-code'. Jul 23 15:09:04 r710-bbb30-2 systemd[1]: Failed to start Uncomplicated firewall.

If you'd run sudo ufw disable command, everything works fine again, except that you don't have firewall enabled. To fix it, reset UFW: sudo ufw reset

And reapply rules from enableUFWRules(): sudo ufw allow OpenSSH sudo ufw allow 'Nginx Full' sudo ufw allow 16384:32768/udp sudo ufw allow 3478

Enable UFW: sudo ufw enable

And everything will work fine, after reboot as well. Surprisingly enough, I haven't faced such problem at VPS powered by one of the hosting providers. Internal network configuration matters? I wasn't able to drill it down to propose patch to fix it in BBB installation script, would be happy to perform necessary checks at problematic VPS, if there would be volunteers.

renatus avatar Jul 25 '25 17:07 renatus

I was able to track down root cause of this issue, there is a known bug in Ubuntu 22.04; 20.04 and, hence, corresponding BBB versions are not affected: https://bugs.launchpad.net/ufw/+bug/1987227 There is a conflict between UFW and iptables-persistent / netfilter-persistent. Yet when you enable UFW alongside iptables-persistent / netfilter-persistent, everything works well, but after reboot with UFW enabled, rules from /etc/iptables/rules.v4 are added but the INPUT->ufw-, FORWARD->ufw-, and OUTPUT->ufw-* hooks are missing. Since UFW default to DROP and the UFW-specific chains are never run, the network connectivity breaks because every single package is dropped. This is the reason everything works well immediately after BBB installation, but breaks after first reboot.

Why not everyone using BBB install script with -w option faces this issue? Because iptables-persistent package is not always being installed, network configuration really matters, as I've previously suspected:

# The turn server will always try to connect to the BBB server's public IP address,
# so if NAT is in use, add an iptables rule to adjust the destination IP address
# of UDP packets sent from the turn server to FreeSWITCH.
if [ -n "$INTERNAL_IP" ]; then
  need_pkg iptables-persistent
  iptables -t nat -A OUTPUT -p udp -s "$INTERNAL_IP" -d "$IP" -j DNAT --to-destination "$INTERNAL_IP"
  netfilter-persistent save
fi

I was able to circumvent core bug modifying BBB installation script so that iptables-persistent is only being installed if UFW is NOT being installed. If UFW is being installed, I'm trying to mimic iptables-persistent functionality with UFW. For me that works very well. I've created pull request: https://github.com/bigbluebutton/bbb-install/pull/795 That's my first pull request for BBB project, would be glad to modify it as needed, if I've missed something, any suggestions are more than welcome.

renatus avatar Aug 15 '25 18:08 renatus