Feature request: Handling API timeouts

[OBenned]

Is your feature request related to a problem? Please describe. We are currently testing your sync script as we would really like to replace our old sync script. But we are currently running into a problem with timeouts both on our old script and this version. Im pretty sure that the problem however is in our end. When running the script we are seeing that most of the time (90 %) there is a timeout to our netxbox API. I have a pretty clear idea that this is the result of gunicorn workers restarting as per the recommended setup from netbox.

Describe the solution you'd like The temp fix i made in out old script was to retry the webrequest to the API if it would fail or timeout. Maybe there could some similar functionality here.

Describe alternatives you've considered N/A

Additional context I don't know python at all, so i don't really have any ideas on how this could be handled, but i would be more than happy to test. I see in #78 that there's a suggestion to implement async sync, maybe this could also help this problem

[synackray]

Hi @OBenned! I can most definitely look into adding a requests retry mechanism and set something like a max retries variable before the script considers a fail condition. I will focus on the feature during the implementation of #78.

[OBenned]

Sounds great, i would be more than happy to test it when the time comes

[bb-Ricardo]

Also running into the issue on pretty much every Sync.

Here is my temporary solution:

diff --git a/run.py b/run.py
index 23601b2..f588999 100644
--- a/run.py
+++ b/run.py
@@ -970,6 +970,27 @@ class NetBoxHandler:
         return result

     def request(self, req_type, nb_obj_type, data=None, query=None, nb_id=None):
+
+        max_retry_attempts = 3
+
+        for _ in range(max_retry_attempts):
+
+            try:
+                result = self.single_request(req_type, nb_obj_type, data, query, nb_id)
+            except (SystemExit, ConnectionError, requests.exceptions.ConnectionError,
+                requests.exceptions.ReadTimeout):
+                log.warning("Request failed, trying again.")
+                continue
+            else:
+                break
+        else:
+            raise SystemExit(
+                    log.critical("Giving up after %d retries.", max_retry_attempts)
+                    )
+
+        return result
+
+    def single_request(self, req_type, nb_obj_type, data=None, query=None, nb_id=None):
         """
         HTTP requests and exception handler for NetBox

Not sure when you're planning to merge #78 as it would be directly affected. I could just do a PR against your current Master but then it wouldn't merge cleanly.

What are your recommendations?