Partitioner sticks in "allocating" if locks are already held

[philipcristiano]

I'm working with the partitioner recipe and have seen some issues in production with the partitioner getting stuck in state "allocating". The debug logs show kazoo repeatedly attempting to acquire and release a lock. This seems to occur when the partitioners use differing sets of items to partition, which makes sense. In this case though I would expect the partition to fail if locks cannot be acquired.

From looking around I think it's related to the non-blocking acquire of the lock https://github.com/python-zk/kazoo/blob/f055dc08d1c24e0c95610852ce99ad38e0ab2688/kazoo/recipe/partitioner.py#L326 which returns False instead of raising an exception.

I will attempt to update it to run assuming that a failing lock means the partition acquisition failed, hopefully causing all processes to clean up and attempt again.