[feature] Implement Celery task for WHOIS fetch based on device last IP

[DragnEmperor]

After closing of #1032, we need to implement celery task which will trigger whenever last_ip changes and is public. The task will be responsible for fetching the required details as per the approach finalized for #1032. The details will then update/create the WHOIS details of that device

Dependencies

• [x] https://github.com/openwisp/openwisp-controller/issues/1032

[DragnEmperor]

Following are the items which I think we can discuss before implementing this to have more clarity:

• Should we add a retry mechanism if task fails?
• Are we providing a way for user to know if the task has succeeded or not? Or if it fails then let the user know of the reason or something that might help in debugging?

[devkapilbansal]

Should we add a retry mechanism if task fails?

Definitely, this retry mechanism should be kept configurable

Are we providing a way for user to know if the task has succeeded or not? Or if it fails then let the user know of the reason or something that might help in debugging?

Probably, we can add a graph for this.

[DragnEmperor]

Should we add a retry mechanism if task fails?

Definitely, this retry mechanism should be kept configurable

I am thinking of something like this, adding this as base class to the celery task:

class BaseRetryTask(Task):
    autoretry_for = (Exception,)    # can be APIException as per requirement
    max_retries = 3
    retry_backoff = 5 * 60      # total period of retrying, i think the difference between retries is based 
                                # on jitter example here for 300second: it will retry 3 times after say 
                                # 90seconds, 170seconds, 40seconds (random). 
                                # this is done to reduce load on celery and let other tasks also run

More information : https://docs.celeryq.dev/en/latest/userguide/tasks.html#Task.autoretry_for

Are we providing a way for user to know if the task has succeeded or not? Or if it fails then let the user know of the reason or something that might help in debugging?

Probably, we can add a graph for this.

I was thinking of sending a notification to user of the failed task, and as for debugging exposing and endpoint that fetches task status. Something like :

class TaskStatusView(APIView):
    def get(self, request, taskID):
        result = AsyncResult(str(taskID))
        payload = {"taskID": taskID, "status": result.status}
        if result.successful():
            payload["result"] = result.result
        elif result.failed() and hasattr(result, "result"):
            payload["result"] = str(result.result)
        return Response(payload)

As for the graph, do you mean visualizing a chart or something like that for showing success vs failed tasks? Sorry if I got that wrong.

[pandafy]

Are we providing a way for user to know if the task has succeeded or not? Or if it fails then let the user know of the reason or something that might help in debugging?

Probably, we can add a graph for this.

I was thinking of sending a notification to user of the failed task, and as for debugging exposing and endpoint that fetches task status. Something like :

class TaskStatusView(APIView): def get(self, request, taskID): result = AsyncResult(str(taskID)) payload = {"taskID": taskID, "status": result.status} if result.successful(): payload["result"] = result.result elif result.failed() and hasattr(result, "result"): payload["result"] = str(result.result) return Response(payload) As for the graph, do you mean visualizing a chart or something like that for showing success vs failed tasks? Sorry if I got that wrong.

I am not following this. We decided to send a notification to the user for creation of a fuzzy location. Maybe, we can send a web notification if the task fails?

For debugging, the users (OpenWISP managers) should be able to lookup the logs on the server.

[DragnEmperor]

Are we providing a way for user to know if the task has succeeded or not? Or if it fails then let the user know of the reason or something that might help in debugging?

Probably, we can add a graph for this.

I was thinking of sending a notification to user of the failed task, and as for debugging exposing and endpoint that fetches task status. Something like : class TaskStatusView(APIView): def get(self, request, taskID): result = AsyncResult(str(taskID)) payload = {"taskID": taskID, "status": result.status} if result.successful(): payload["result"] = result.result elif result.failed() and hasattr(result, "result"): payload["result"] = str(result.result) return Response(payload) As for the graph, do you mean visualizing a chart or something like that for showing success vs failed tasks? Sorry if I got that wrong.

I am not following this. We decided to send a notification to the user for creation of a fuzzy location. Maybe, we can send a web notification if the task fails?

For debugging, the users (OpenWISP managers) should be able to lookup the logs on the server.

This is a sample code for returning the status of task and its result if task failed or succeeds. Yes, we don't need this if server logs can be accessed.

I think I can proceed with an implementation of the failure mechanisms. Thanks @devkapilbansal @pandafy