i/o timeout error on very intensive benchmarks

[filipecosta90]

Reference: https://circleci.com/api/v1.1/project/github/RediSearch/RediSearch/61497/output/115/0?file=true&allocation-id=63e39a2ad75eca65ca0cb81e-0-build%2F4B7B1FD0

0BB/s  0BB/s\n', '2023/02/08 13:49:35 read tcp 10.3.0.239:39968->10.3.0.65:6379: i/o timeout\n', '2023/02/08 13:49:35 read tcp 10.3.0.239:39886->10.3.0.65:6379: i/o timeout\n', '2023/02/08 13:49:35 read tcp 10.3.0.239:39930->10.3.0.65:6379: i/o timeout\n']

[fcostaoliveira]

issue appeared again in https://productionresultssa9.blob.core.windows.net/actions-results/4cca77c0-e46f-4d80-b11b-3a54a601a1ff/workflow-job-run-99bd9abd-bfc6-5635-8bb3-28a42da672ab/logs/job/job-logs.txt?rsct=text%2Fplain&se=2024-09-18T14%3A44%3A37Z&sig=RX8Mn4v40SwS5t8n6PrqIsTSvZ0%2Fz5suO0wnw%2Fp7AGI%3D&ske=2024-09-19T01%3A37%3A40Z&skoid=ca7593d4-ee42-46cd-af88-8b886a2f84eb&sks=b&skt=2024-09-18T13%3A37%3A40Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2024-05-04&sp=r&spr=https&sr=b&st=2024-09-18T14%3A34%3A32Z&sv=2024-05-04

2024-09-18T14:12:13.3198760Z 2024-09-18 14:12:13,319 INFO Connecting to remote server 18.219.137.1
2024-09-18T14:14:27.9168966Z CRITICAL:root:Some unexpected exception was caught during remote work on test named search-ftsb-5500K-docs-union-iterators-q2. Failing test....
2024-09-18T14:14:27.9171380Z 2024-09-18 14:14:27,916 CRITICAL Some unexpected exception was caught during remote work on test named search-ftsb-5500K-docs-union-iterators-q2. Failing test....
2024-09-18T14:14:27.9172910Z CRITICAL:root:<class 'TimeoutError'>
2024-09-18T14:14:27.9173728Z 2024-09-18 14:14:27,916 CRITICAL <class 'TimeoutError'>
2024-09-18T14:14:27.9179689Z 
2024-09-18T14:14:27.9182750Z 100%|██████████| 1/1 [04:23<00:00, 263.67s/benchmarks]
2024-09-18T14:14:27.9184873Z     Complete percent: 100.00 %setup writes/sec          writes/sec         updates/sec           reads/sec    cursor reads/sec         deletes/sec     current ops/sec           total ops             TX BW/sRX BW/s
2024-09-18T14:14:27.9186577Z 
2024-09-18T14:14:27.9187609Z 0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)                   0               0BB/s  0BB/s
2024-09-18T14:14:27.9189486Z 0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)                   0               0BB/s  0BB/s
2024-09-18T14:14:27.9191318Z 0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)                   0               0BB/s  0BB/s
2024-09-18T14:14:27.9193139Z 0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)                   0               0BB/s  0BB/s
2024-09-18T14:14:27.9194957Z 0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)                   0               0BB/s  0BB/s
2024-09-18T14:14:27.9196750Z 0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)                   0               0BB/s  0BB/s
2024-09-18T14:14:27.9198584Z 0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)                   0               0BB/s  0BB/s
2024-09-18T14:14:27.9200385Z 0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)                   0               0BB/s  0BB/s
2024-09-18T14:14:27.9202618Z 0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)           0 (0.000)                   0               0BB/s  0BB/s
2024-09-18T14:14:27.9204705Z 2024/09/18 14:12:12 Error preparing for redisearch ingestion, while creating new pool. error = dial tcp 10.3.0.56:6379: i/o timeout
2024-09-18T14:14:27.9207015Z 2024/09/18 14:12:12 Error preparing for redisearch ingestion, while creating new pool. error = dial tcp 10.3.0.56:6379: i/o timeout
2024-09-18T14:14:27.9208776Z ------------------------------------------------------------
2024-09-18T14:14:27.9209728Z Traceback (most recent call last):
2024-09-18T14:14:27.9211495Z   File "/usr/local/lib/python3.9/dist-packages/redisbench_admin/run_remote/run_remote.py", line 511, in run_remote_command_logic
2024-09-18T14:14:27.9213063Z     ) = remote_db_spin(
2024-09-18T14:14:27.9214588Z   File "/usr/local/lib/python3.9/dist-packages/redisbench_admin/run_remote/remote_db.py", line 351, in remote_db_spin
2024-09-18T14:14:27.9216062Z     db_error_artifacts(
2024-09-18T14:14:27.9217627Z   File "/usr/local/lib/python3.9/dist-packages/redisbench_admin/run_remote/remote_db.py", line 406, in db_error_artifacts
2024-09-18T14:14:27.9219138Z     execute_remote_commands(
2024-09-18T14:14:27.9220712Z   File "/usr/local/lib/python3.9/dist-packages/redisbench_admin/utils/remote.py", line 142, in execute_remote_commands
2024-09-18T14:14:27.9222471Z     c = connect_remote_ssh(port, private_key, server_public_ip, username)
2024-09-18T14:14:27.9224629Z   File "/usr/local/lib/python3.9/dist-packages/redisbench_admin/utils/remote.py", line 165, in connect_remote_ssh
2024-09-18T14:14:27.9226365Z     c.connect(hostname=server_public_ip, port=port, username=username, pkey=k)
2024-09-18T14:14:27.9227628Z   File "/usr/local/lib/python3.9/dist-packages/paramiko/client.py", line 386, in connect
2024-09-18T14:14:27.9228382Z     sock.connect(addr)
2024-09-18T14:14:27.9228847Z TimeoutError: [Errno 110] Connection timed out
2024-09-18T14:14:27.9229492Z ------------------------------------------------------------
2024-09-18T14:14:27.9230213Z   0%|          | 0/1 [00:00<?, ?benchmarks/s][A
2024-09-18T14:14:27.9230876Z 100%|██████████| 1/1 [04:23<00:00, 263.67s/benchmarks]