companion icon indicating copy to clipboard operation
companion copied to clipboard

[BUG] Companion v3.4 UI slow to load & Satellite connections drop/reconnect

Open JuddiP opened this issue 1 year ago • 6 comments

Is this a bug in companion itself or a module?

  • [X] I believe this to be a bug in companion

Is there an existing issue for this?

  • [x] I have searched the existing issues

Describe the bug

We have Companion running in a virtualised environment, on both Windows 10 and Ubuntu 24.04 LTS hosts. Since updating to v3.4 we have noticed considerable delay when initially loading the GUI (specifically Buttons tab). When the GUI is reloaded, all connected Satellite surfaces are bounced (disconnect & reconnect). This occurs when the GUI is reloaded on both a local network PC, and via VPN.

Network usage of the webclient has been observed to spike to +900Mbps for 2-3 seconds while the GUI loads (local network PC).

Webclient PC image

Behavior is better on Windows 10 than on Ubuntu -- the Satellite surfaces do not drop.

This behavior was not observed when running v3.3.1, so I suspect some change in Companion core is the root cause.

Steps To Reproduce

  1. Ensure Companion is on v3.4
  2. Connect a large number of connections
  3. Reload GUI while on any tab
  4. All Satellite connections bounce

Expected Behavior

While on v3.3.1

  1. Connect a large number of connections
  2. Reload GUI while on any tab
  3. GUI reloads promptly, Satellite surfaces do not bounce

Environment (please complete the following information)

- OS:Windows 10/Ubuntu 24.04 LTS
- Browser:Chrome 128.0.6613.138 (Official Build) (64-bit)
- Companion Version: 3.4
- VM CPU Cores: 6
- VM RAM: 12GB


- Connections: 85 (total), 56 (active)
- Satellite Surfaces: 7
- Emulators: 5

Additional context

No response

JuddiP avatar Sep 17 '24 03:09 JuddiP

Reset data counters on the Companion host PC and completed 1 reload of the webGUI from a remote client on the local network. The webclient is pulling 125MB of data when it loads - image

JuddiP avatar Sep 17 '24 23:09 JuddiP

could you send me your 3.3.1 config to [email protected] (make sure to remove any api keys for cloud services first) ? I can't think of anything that would cause this, nothing from the changelog jumps out to me as being a candidate for causing this. so I think this is going to require some cpu profiling to figure out what it is doing and see what could be causing this

Julusian avatar Sep 18 '24 22:09 Julusian

Thanks for responding to this, @Julusian. I'll get the 3.3.1 as well as the 3.4 config to you later today (with private keys stripped out).

Further info on our troubleshooting and environment -

  • Rolled back to our Windows VM (v3.4) to see if it behaved better than Linux. Satellite connections don't drop when the web page is reloaded, and it seems to be slightly faster. Same CPU & RAM allocations.
  • VMs are configured in Hyper-V on a Windows Server 2019 host (Dell PowerEdge R640).
  • If needed we can revert to an older checkpoint (v3.3.1) once free from production.
  • Grid layout: 4x32 - surfaces across 3 control rooms are offset to use horizontal zones on each page.
  • We have at least 3 cloud service connections running (iCap, LEXI, LiveU).
  • Periodically we will have connections to PTZ cameras over the internet (LiveU IP PIPE).
  • We have a lot of custom graphics on buttons, but where possible they have been made into advanced feedbacks to increase efficiency/lower required storage.

JuddiP avatar Sep 18 '24 22:09 JuddiP

Emailed those files through -- cheers!

JuddiP avatar Sep 19 '24 00:09 JuddiP

i have experienced the same, but have not been able to confirm why its happening, also running on v3.4.0. i have about 3 Individual Satellites with each with its own Stream Deck's running on Windows 11 PC's, and had a few times where all 3 disconnected simultaneously. They don't reconnect again unless i pull the Stream Deck's USB cable and plug it back in or if i close the Satellite Apps.

When i logged into the web client i saw it created a bunch of Offline Surfaces, with the ID's set to Chinese characters.

Bambii556 avatar Oct 14 '24 17:10 Bambii556

I really wish the UI would auto reconnect - both satellite and web emulator!

AJolly avatar Nov 14 '24 07:11 AJolly