intermittent test_cleanup_containers() failing

[jpopelka]

For example here:

11:47:45 =================================== FAILURES ===================================
11:47:45 ___________________________ test_cleanup_containers ____________________________
11:47:45 
11:47:45     def test_cleanup_containers():
11:47:45         with DockerBackend(logging_level=logging.DEBUG, cleanup=[CleanupPolicy.CONTAINERS]) as backend:
11:47:45             # cleaning up from previous runs
11:47:45             backend.cleanup_containers()
11:47:45     
11:47:45             client = get_client()
11:47:45             container_sum = len(client.containers(all=True))
11:47:45             image = DockerImage(FEDORA_MINIMAL_REPOSITORY, tag=FEDORA_MINIMAL_REPOSITORY_TAG)
11:47:45             command = ["ls"]
11:47:45             additional_opts = ["-i", "-t"]
11:47:45     
11:47:45             for i in range(3):
11:47:45                 image.run_via_binary(command=command, additional_opts=additional_opts)
11:47:45     
11:47:45 >           assert container_sum+3 == len(client.containers(all=True))
11:47:45 E           AssertionError: assert (23 + 3) == 25
11:47:45 E            +  where 25 = len([{'Command': 'ls', 'Created': 1534247099, 'HostConfig': {'NetworkMode': 'default'}, 'Id': '2b19224e1131020e63846d3943a...tConfig': {'NetworkMode': 'host'}, 'Id': 'd6f4675524701350b1697c9d32967d29e3e80723dbfb6f1e5431977cb9777c11', ...}, ...])
11:47:45 E            +    where [{'Command': 'ls', 'Created': 1534247099, 'HostConfig': {'NetworkMode': 'default'}, 'Id': '2b19224e1131020e63846d3943a...tConfig': {'NetworkMode': 'host'}, 'Id': 'd6f4675524701350b1697c9d32967d29e3e80723dbfb6f1e5431977cb9777c11', ...}, ...] = <bound method APIClient.containers of <docker.api.client.APIClient object at 0x7f197f853fd0>>(all=True)
11:47:45 E            +      where <bound method APIClient.containers of <docker.api.client.APIClient object at 0x7f197f853fd0>> = <docker.api.client.APIClient object at 0x7f197f853fd0>.containers
11:47:45 
11:47:45 tests/integration/test_backend.py:41: AssertionError

[TomasTomecek]

I suspect this is a race: 2 concurrent test runs affect each other.

My proposal: short term - remove the assert with precise numbers, long term - enable numbers back and allow concurrent test runs (I feel like this will require changes to cleanup mechanism). Over.

[dhodovsk]

Add unique identifier for conu LABEL in every instance of DockerBackend.

Or store all created containers in a list. Similarly as in https://github.com/user-cont/conu/blob/master/conu/backend/k8s/backend.py#L62