Add locking mechanism to Turbinia Server

[aarontp]

This is so that multiple Turbinia Servers don't stomp on each other when using the same PubSub queue for scheduling tasks, etc.

[aarontp]

I chatted with @alimez about this a bit, and similar to https://github.com/google/turbinia/issues/87, I think the first solution here is to periodically write heartbeat/timestamp information into datastore keyed by hostname, and the server can check whether there is another server started with the same instance-id/hostname within the last N minutes, and refuse to start if it finds it.

I think this will solve most cases, but there will still be a race condition here. Another check we could perform here is to generate a unique id on the server, and pass it through the Tasks, and if a TaskResult comes back to a server with a server id that does not match, we can handle it appropriately then (possibly by exiting). Alternately, an easier solution could be to just have the server continue to do checks of the heartbeat data as it's running, and not just at server start time (maybe at the same time as when it writes its own data).

[jleaniz]

@aarontp Is this still something we want to look into with Celery?

[aarontp]

I don't think we really need this anymore given the way that Celery works.