Orphaned workhorse recreates redis key of the killed worker with TTL `-1`

[kusaku]

trafficstars

When a worker is terminated with kill -9 (for example, suprevisor can terminate workers with SIGKILL), the workhorse will continue running.

Workhorse may finish or fail. In both cases it will execute some stats methods:

• Worker.increment_successful_job_count()
• Worker.increment_failed_job_count()
• Worker.increment_total_working_time()

These methods will execute redis HINCRBY and HINCRBYFLOAT on a non-existent key of terminated worker (redis key of worker is deleted by TTL), for absent key redis will set the standard ttl -1and that key will become immortal.`

A check could be added to see if the worker still exists before updating the stats.

[selwin]

Agreed. However, the checks shouldn't be done everytime the keys are incremented as this will incur one extra Redis operation for every job that finishes.

The check should only be performed when increment_successful_job_count() or increment_failed_job_count() returns 1 (meaning it's possibly a new key).

Do you mind opening a PR for this?