Retry Worker Side Timeouts

[wajihyassine]

Celery has the ability to retry Tasks and only retry an X max amount times before failing.

We'd benefit from a Worker side timeout to at least retry a timed out job once given times where there is high load and short timeouts and important jobs we want to make sure run.

This can be configurable either by a worker side timeout or a job based to Task timeout. A value of zero can be given to not retry a Task if it was timed out.

More info around Celery Retries here: https://docs.celeryq.dev/en/stable/userguide/tasks.html#retrying

[aarontp]

Related issue: https://github.com/google/turbinia/issues/21