pymongo.errors.OperationFailure: No keys found for HMAC that is valid for time

[JustAnotherVeryNormalDeveloper]

I have a program that worked all the time. Another team made multiple changement on the servers hardware and IP. Now it works 90% of the time. 10% of the time, the script fail. I added 5 retries with time.sleep. I upgraded pymongo version: 4.3.3 -> 4.10.1 It still happens (it feels to happen a little bit less but I have no proper way to test it easely).

Have you advices on how to improve this ? what this error means precisely? I am running out of ideas. :/

TRACE:

File \"/app/src/databases/servers/mongo_server_common.py\", line 18, in __init__", "    MongoServer.__init__(self, mongo_uri)", "  File \"/app/src/databases/servers/mongo_server.py\", line 14, in __init__", "    self._client = self.createMongoClient()", "  File \"/app/src/databases/servers/mongo_server.py\", line 34, in createMongoClient", "    client.server_info() # Test connection", "  File \"/usr/local/lib/python3.9/site-packages/pymongo/synchronous/mongo_client.py\", line 2125, in server_info", "    self.admin.command(", "  File \"/usr/local/lib/python3.9/site-packages/pymongo/_csot.py\", line 119, in csot_wrapper", "    return func(self, *args, **kwargs)", "  File \"/usr/local/lib/python3.9/site-packages/pymongo/synchronous/database.py\", line 930, in command", "    return self._command(", "  File \"/usr/local/lib/python3.9/site-packages/pymongo/synchronous/database.py\", line 770, in _command", "    return conn.command(", "  File \"/usr/local/lib/python3.9/site-packages/pymongo/synchronous/helpers.py\", line 45, in inner", "    return func(*args, **kwargs)", "  File \"/usr/local/lib/python3.9/site-packages/pymongo/synchronous/pool.py\", line 538, in command", "    return command(", "  File \"/usr/local/lib/python3.9/site-packages/pymongo/synchronous/network.py\", line 218, in command", "    helpers_shared._check_command_response(", "  File \"/usr/local/lib/python3.9/site-packages/pymongo/helpers_shared.py\", line 247, in _check_command_response", "    raise OperationFailure(errmsg, code, response, max_wire_version)", "pymongo.errors.OperationFailure: No keys found for HMAC that is valid for time: { ts: Timestamp(xxx, 2) } with id: xxx, full error: {'ok': 0.0, 'errmsg': 'No keys found for HMAC that is valid for time: { ts: Timestamp(xxx, 2) } with id: xxx', 'code': 211, 'codeName': 'KeyNotFound', '$clusterTime': {'clusterTime': Timestamp(xxx, 1), 'signature': {'hash': xxx', 'keyId': xxx}}, 'operationTime': Timestamp(xxx, 1)}"]}