docker stop analyticsdb - gets stuck indefinitely

[sangupta01]

Using build 3023 but saw with 3018 too. Seen this issue using docker builds from 10.84.34.155 and for 3018/3023 pvt docker images/registry.

Not able to stop containers. "docker kill" also gets stuck.

• no errors seen in "docker logs " during install using ansible.
• no errors seen in "docker logs -f " during "docker stop ..." This is seen mostly with analyticsdb and controller containers. Sometimes rebooting the node multiple times and trying "docker stop ..." works. logging into the container and stopping services (kafka, database-manager) didnt help.

2017-01-13 06:10:11,483 INFO exited: kafka (exit status 1; not expected) FAILED - RETRYING: TASK: contrail/analyticsdb : Wait till config api server answers (121 retries left). 2017-01-13 06:10:12,486 INFO spawned: 'kafka' with pid 2833 FAILED - RETRYING: TASK: contrail/analyticsdb : Wait till config api server answers (120 retries left). 2017-01-13 06:10:13,657 INFO stopped: kafka (exit status 1) FAILED - RETRYING: TASK: contrail/analyticsdb : Wait till config api server answers (119 retries left). ... FAILED - RETRYING: TASK: contrail/analyticsdb : Wait till config api server answers (107 retries left). 2017-01-13 06:10:28,933 INFO stopped: contrail-database-nodemgr (terminated by SIGTERM)

At this point, there is no further dump on the screen even if I do CTRL-C and "docker stop/kill" in another screen. "docker ps -a" still shows STATUS as UP.

Even waiting till 0 retries left and then failed, "docker stop ..." gets stuck. Trying to login shows below error. root@a1s42:~# docker exec -it analyticsdb /bin/bash rpc error: code = 13 desc = invalid header field value "oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:83: executing setns process caused \"exit status 16\""\n"