Supporting graceful worker exits (alternatives to CELERYD_MAX_TASKS_PER_CHILD), eg for leaky workers

[samv]

Hi,

I was wondering if it would be possible to have a custom hook or a special exception class that would allow a worker to decide at runtime that it should not continue.

The application is to allow workers which detect they've allocated too much memory since they were started to exit gracefully. I find this a much more useful measure than the crude "max requests per child" approach, and keeps system memory useful and active. Currently I'm just raising SystemExit(), which seems to be interpreted as an error by billiard, making it raise WorkerLostError.

If you've got no objection, please let me know how I can help!

Cheers, Sam.