list completed tasks more efficiently

[timodonnell]

We currently detect what tasks have completed by issuing a bucket list request with the prefix set to the common prefix of the result names for all running tasks. This is inefficient when the common prefix includes a large number of tasks, which can happen when our running tasks happen to have task numbers like 4999, 5000, 5001, etc. As there is no common prefix to these task numbers, all tasks get selected. I think there may be ways to be more clever about task naming so we limit how many tasks we request.

Current output illustrating the multiple requests:

2016-12-21 20:57:59.252 INFO common - check_call: Running: (['kubectl', 'create', '-f', '/tmp/kubeface-kubernetes-hammerla-odonnt02-2016-12-14-23:31:33-16c46283::54673j3z9iik1.json'],) {}
pod "hammerla-odonnt02-2016-12-14-23-31-33-16c46283--54673" created
2016-12-21 20:58:00.40 INFO discovery - method: URL being requested: POST https://www.googleapis.com/upload/storage/v1/b/kubeface/o?alt=json&uploadType=multipart
2016-12-21 20:58:00.401 INFO discovery - method: URL being requested: POST https://www.googleapis.com/upload/storage/v1/b/kubeface/o?alt=json&uploadType=multipart
2016-12-21 20:58:00.543 INFO discovery - method: URL being requested: GET https://www.googleapis.com/storage/v1/b/kubeface/o?prefix=result%3A%3Ahammerla-odonnt02-2016-12-14-23%3A31%3A33-16c46283%3A%3A5&fields=nextPageToken%2Citems%28name%29&alt=json&maxResults=100000
2016-12-21 20:58:00.785 INFO discovery - methodNext: URL being requested: list_next https://www.googleapis.com/storage/v1/b/kubeface/o?prefix=result%3A%3Ahammerla-odonnt02-2016-12-14-23%3A31%3A33-16c46283%3A%3A5&fields=nextPageToken%2Citems%28name%29&alt=json&maxResults=100000&pageToken=Cj1yZXN1bHQ6OmhhbW1lcmxhLW9kb25udDAyLTIwMTYtMTItMTQtMjM6MzE6MzMtMTZjNDYyODM6OjUwODk4
2016-12-21 20:58:00.970 INFO discovery - methodNext: URL being requested: list_next https://www.googleapis.com/storage/v1/b/kubeface/o?prefix=result%3A%3Ahammerla-odonnt02-2016-12-14-23%3A31%3A33-16c46283%3A%3A5&fields=nextPageToken%2Citems%28name%29&alt=json&maxResults=100000&pageToken=Cj1yZXN1bHQ6OmhhbW1lcmxhLW9kb25udDAyLTIwMTYtMTItMTQtMjM6MzE6MzMtMTZjNDYyODM6OjUxNzk4
2016-12-21 20:58:01.209 INFO discovery - methodNext: URL being requested: list_next https://www.googleapis.com/storage/v1/b/kubeface/o?prefix=result%3A%3Ahammerla-odonnt02-2016-12-14-23%3A31%3A33-16c46283%3A%3A5&fields=nextPageToken%2Citems%28name%29&alt=json&maxResults=100000&pageToken=Cj1yZXN1bHQ6OmhhbW1lcmxhLW9kb25udDAyLTIwMTYtMTItMTQtMjM6MzE6MzMtMTZjNDYyODM6OjUyNjk4
2016-12-21 20:58:01.452 INFO discovery - methodNext: URL being requested: list_next https://www.googleapis.com/storage/v1/b/kubeface/o?prefix=result%3A%3Ahammerla-odonnt02-2016-12-14-23%3A31%3A33-16c46283%3A%3A5&fields=nextPageToken%2Citems%28name%29&alt=json&maxResults=100000&pageToken=Cj1yZXN1bHQ6OmhhbW1lcmxhLW9kb25udDAyLTIwMTYtMTItMTQtMjM6MzE6MzMtMTZjNDYyODM6OjUzNjMy
2016-12-21 20:58:31.785 INFO discovery - method: URL being requested: GET https://www.googleapis.com/storage/v1/b/kubeface/o?prefix=result%3A%3Ahammerla-odonnt02-2016-12-14-23%3A31%3A33-16c46283%3A%3A5&fields=nextPageToken%2Citems%28name%29&alt=json&maxResults=100000
2016-12-21 20:58:32.33 INFO discovery - methodNext: URL being requested: list_next https://www.googleapis.com/storage/v1/b/kubeface/o?prefix=result%3A%3Ahammerla-odonnt02-2016-12-14-23%3A31%3A33-16c46283%3A%3A5&fields=nextPageToken%2Citems%28name%29&alt=json&maxResults=100000&pageToken=Cj1yZXN1bHQ6OmhhbW1lcmxhLW9kb25udDAyLTIwMTYtMTItMTQtMjM6MzE6MzMtMTZjNDYyODM6OjUwODk4
2016-12-21 20:58:32.331 INFO discovery - methodNext: URL being requested: list_next https://www.googleapis.com/storage/v1/b/kubeface/o?prefix=result%3A%3Ahammerla-odonnt02-2016-12-14-23%3A31%3A33-16c46283%3A%3A5&fields=nextPageToken%2Citems%28name%29&alt=json&maxResults=100000&pageToken=Cj1yZXN1bHQ6OmhhbW1lcmxhLW9kb25udDAyLTIwMTYtMTItMTQtMjM6MzE6MzMtMTZjNDYyODM6OjUxNzk4
2016-12-21 20:58:32.549 INFO discovery - methodNext: URL being requested: list_next https://www.googleapis.com/storage/v1/b/kubeface/o?prefix=result%3A%3Ahammerla-odonnt02-2016-12-14-23%3A31%3A33-16c46283%3A%3A5&fields=nextPageToken%2Citems%28name%29&alt=json&maxResults=100000&pageToken=Cj1yZXN1bHQ6OmhhbW1lcmxhLW9kb25udDAyLTIwMTYtMTItMTQtMjM6MzE6MzMtMTZjNDYyODM6OjUyNjk4
2016-12-21 20:58:32.737 INFO discovery - methodNext: URL being requested: list_next https://www.googleapis.com/storage/v1/b/kubeface/o?prefix=result%3A%3Ahammerla-odonnt02-2016-12-14-23%3A31%3A33-16c46283%3A%3A5&fields=nextPageToken%2Citems%28name%29&alt=json&maxResults=100000&pageToken=Cj1yZXN1bHQ6OmhhbW1lcmxhLW9kb25udDAyLTIwMTYtMTItMTQtMjM6MzE6MzMtMTZjNDYyODM6OjUzNjMy