clearwater-live-test SUBSCRIBE timeout tests unreliable against Docker deployment

[graemerobertson]

We create a deployment by running the following commands...

docker build --no-cache -t clearwater/base base
docker-compose -f minimal-distributed.yaml up --force-recreate -d
docker-compose -f minimal-distributed.yaml scale memcached=2 chronos=2 cassandra=2

We then sleep for 4 minutes and run the live tests against our new deployment. They all pass, except the SUBSCRIBE - Subscription timeout and SUBSCRIBE - Registration timeout tests which timeout "waiting for NOTIFY". It seems to be the expiry NOTIFY that we don't receive.

We run this regularly using Jenkins (the live-test-docker job), and these tests seem to fail most of the time. We collect some diags, but they're currently not very useful. I've excluded these tests from that job now, so we should turn them back on when this is fixed.

In order to repro, I'd just run through the live-test-docker job manually (with the tests not excluded!).