[BUG] - Flaky local integration tests failing with paramiko.ssh_exception.AuthenticationException

[marcelovilla]

Describe the bug

I've noticed that sporadically, our local integration tests might fail with the following error:

------------------------------ Captured log setup ------------------------------
2024-10-17 06:18:35      INFO 1944 transport: Connected (version 2.0, client AsyncSSH_2.13.2)
2024-10-17 06:18:35      INFO 1944 transport: Auth banner: b'Starting your server...'
2024-10-17 06:18:36      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:36      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:37      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:37      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:38      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:38      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:39      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:39      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:40      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:40      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:41      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:41      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:42      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:42      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:43      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:43      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:44      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:44      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:45      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:46      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:46      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:47      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:47      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:48      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:48      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:49      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:49      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:50      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:50      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:51      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:51      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:52      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:52      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:53      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:53      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:54      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:54      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:55      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:55      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:56      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:56      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:57      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:57      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:58      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:58      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:59      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:19:00      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:19:00      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:19:01      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:19:01      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:19:02      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:19:02      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:19:03      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:19:03      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:19:04      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:19:04      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:19:05      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:19:05      INFO 1944 transport: Auth banner: b'failed to start server on time!\n'
2024-10-17 06:19:05      INFO 1944 transport: Authentication (password) failed.
=========================== short test summary info ============================
ERROR tests/tests_deployment/test_jupyterhub_ssh.py::test_simple_jupyterhub_ssh - paramiko.ssh_exception.AuthenticationException: Authentication failed.
==================== 27 passed, 1 error in 73.40s (0:01:13) ====================

Usually a simple re-trigger of the GHA workflow will work. See for example:

• https://github.com/nebari-dev/nebari/actions/runs/11379267190/attempts/1?pr=2773 (failed)
• https://github.com/nebari-dev/nebari/actions/runs/11379267190/attempts/2?pr=2773 (succeeded after re-trigger)

Expected behavior

Local integration tests should not fail because of an authentication error, specially if suggested changes in a PR don't involve that component.

OS and architecture in which you are running Nebari

GHA

How to Reproduce the problem?

This issue happens only sporadically and it's not consistently reproducible.

Command output

No response

Versions and dependencies used.

No response

Compute environment

None

Integrations

No response

Anything else?

No response