8.x - Uncaught exception in ZMQStream callback

[tfmark]

We are in the process of upgrading our Z2JH instance from pre-1.0 to 'latest' (jupyterlab 3.6.3). We encountered an issue where our notebooks either locked up for a period, or ultimately failed to finish executing. I created a post on the Discourse, but ultimately the solution appears to be to downgrade to jupyter_client==7.4.9 - which, after looking at other issues, I don't seem to be the only person facing issues with the 8.x release.

Per the discourse thread, I am unable to reproduce the issue running a standalone jupyterlab instance, and the issue surfaces almost immediately with jupyter_client version 8.

I'm happy to test or provide more logs etc - I just really struggled to find the root cause of the problem.