netplugin
netplugin copied to clipboard
Containers spawned through `docker service` doesn't get DNS names through Contiv IPAM driver
Description
Containers spawned using Contiv network and Contiv IPAM driver fails to get DNS names and host names. Container host name takes container id instead of container names as should be provided by Contiv IPAM driver as part of service discovery feature.
Expected Behavior
Contiv IPAM driver should generated the DNS names for containers while giving IP address to them and use those name as host names for the containers through docker service
create work-flow.
Observed Behavior
Containers are getting names as randomly generated containers IDs
Steps to Reproduce (for bugs)
- Create Contiv network with nw-tag
- Create Docker network using contiv-tag & contiv network driver and IPAM driver
- Spawn containers using
docker service create
using contiv network - Once service gets deployed log-in to the containers and check the hostnames
Your Environment
- netctl version - 1.1.1
- Orchestrator version (e.g. kubernetes, mesos, swarm): UCP/Swarm - 2.2.2
- Operating System and version - RHEL 7.3 Bare Metal cluster nodes
Initial triage -
1). [root@DEE-Ctrl-1 ~]# netctl network ls -a Tenant Network Nw Type Encap type Packet tag Subnet Gateway IPv6Subnet IPv6Gateway Cfgd Tag
tenant-1 contiv-test-1 data vlan 1001 100.100.100.0/24 100.100.100.254 tenant-1-test-1 tenant-2 contiv-test-2 data vlan 1002 200.200.200.0/24 200.200.200.254 tenant-2-test-2
2). [root@DEE-Ctrl-1 ~]# docker network ls NETWORK ID NAME DRIVER SCOPE 701a8b77777f bridge bridge local 0xebqmj20v6s contiv-test-1 contiv/v2plugin:1.1.1 swarm gegwyxkagj7u contiv-test-2 contiv/v2plugin:1.1.1 swarm 799fe436bc46 docker_gwbridge bridge local o6a7ob7q8aqa dtr-ol overlay swarm 61735572149c host host local swcu9ofu7amv ingress overlay swarm gzs3dqjm2eln my-overlay overlay swarm a2295b13a327 none null local
3). RKHARYA-M-H07C:ucp-17.06-B rkharya$ docker service create --name apline-tenant-1 --network contiv-test-1 alpine sleep 100000 xdz75i68l9p4dxv7zvwrm01yp
4). RKHARYA-M-H07C:ucp-17.06-B rkharya$ docker service scale apline-tenant-1=8 apline-tenant-1 scaled to 8
[root@DEE-Ctrl-1 ~]# docker service ls ID NAME MODE REPLICAS IMAGE PORTS 8wqtn92lavk9 ucp-agent-s390x global 0/0 docker/ucp-agent-s390x:2.2.2 i3ttsfu22lcv ucp-agent global 10/10 docker/ucp-agent:2.2.2 vvqbvxt8zdas busybox-tenant-2 replicated 8/8 busybox:latest xaapf3q2n34x ucp-agent-win global 0/0 docker/ucp-agent-win:2.2.2 xdz75i68l9p4 apline-tenant-1 replicated 8/8 alpine:latest
5). RKHARYA-M-H07C:ucp-17.06-B rkharya$ docker service ps apline-tenant-1 ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS 8msxuxk1wlf0 apline-tenant-1.1 alpine:latest DEE-Ctrl-1.cisco.com Running Running about a minute ago e5n36ut19kcu apline-tenant-1.2 alpine:latest DEE-Wrk-1.cisco.com Running Running 36 seconds ago 9qqln1emoz28 apline-tenant-1.3 alpine:latest DEE-Wrk-2.cisco.com Running Running 36 seconds ago gc97stbogy20 apline-tenant-1.4 alpine:latest DEE-DTR-1.cisco.com Running Running 36 seconds ago 2aop3g8x2349 apline-tenant-1.5 alpine:latest DEE-DTR-2.cisco.com Running Running 36 seconds ago khsby8641gvw apline-tenant-1.6 alpine:latest DEE-Ctrl-2.cisco.com Running Running 36 seconds ago u1qbd6gzj9qn apline-tenant-1.7 alpine:latest DEE-DTR-3.cisco.com Running Running 36 seconds ago hzjgcik16cvf apline-tenant-1.8 alpine:latest DEE-Wrk-4.cisco.com Running Running 36 seconds ago
6). RKHARYA-M-H07C:ucp-17.06-B rkharya$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES df679daf7461 alpine:latest "sleep 100000" 10 seconds ago Up 7 seconds DEE-DTR-3.cisco.com/apline-tenant-1.7.u1qbd6gzj9qn2evgm7vukfz16 deb250d36ca1 alpine:latest "sleep 100000" 10 seconds ago Up 6 seconds DEE-Ctrl-2.cisco.com/apline-tenant-1.6.khsby8641gvwcjwzvep8blxf7 e99534004598 alpine:latest "sleep 100000" 10 seconds ago Up 7 seconds DEE-Wrk-1.cisco.com/apline-tenant-1.2.e5n36ut19kcu3ywj0yevyaopf f663d5dd6291 alpine:latest "sleep 100000" 10 seconds ago Up 6 seconds DEE-Wrk-4.cisco.com/apline-tenant-1.8.hzjgcik16cvf15rfrgmq9lm3n 9ced4eec08d1 alpine:latest "sleep 100000" 10 seconds ago Up 7 seconds DEE-Wrk-2.cisco.com/apline-tenant-1.3.9qqln1emoz28un9tunx8alzlx 647fad882b94 alpine:latest "sleep 100000" 10 seconds ago Up 7 seconds DEE-DTR-1.cisco.com/apline-tenant-1.4.gc97stbogy20vmubfe8fddvf9 350756ffb379 alpine:latest "sleep 100000" 10 seconds ago Up 7 seconds DEE-DTR-2.cisco.com/apline-tenant-1.5.2aop3g8x23
7). RKHARYA-M-H07C:ucp-17.06-B rkharya$ docker exec -it df679daf7461 /bin/sh / # cat /etc/hosts 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters 100.100.100.7 df679daf7461
8).
[root@DEE-Ctrl-1 ~]# netctl network inspect contiv-test-1 -t tenant-1
{
"Config": {
"key": "tenant-1:contiv-test-1",
"cfgdTag": "tenant-1-test-1",
"encap": "vlan",
"gateway": "100.100.100.254",
"networkName": "contiv-test-1",
"nwType": "data",
"pktTag": 1001,
"subnet": "100.100.100.0/24",
"tenantName": "tenant-1",
"link-sets": {},
"links": {
"Tenant": {
"type": "tenant",
"key": "tenant-1"
}
}
},
"Oper": {
"allocatedAddressesCount": 9,
"allocatedIPAddresses": "100.100.100.1-100.100.100.9, 100.100.100.254",
"availableIPAddresses": "100.100.100.10-100.100.100.253",
"endpoints": [
{
docker service create has
--hostname
option if that is what you are looking to configure.
we are not looking for docker provided --hostname option. That option gives the container hostname to the first container only. Rest of the containers in the service get the same name as first container.
This issue is to report Contiv IPAM driver's service discovery feature not working correctly with docker service model.
If we use docker run
, legacy way of deploying container, container name specified to docker run command also gets attached to /etc/hosts file inside container correctly by contiv IPAM driver. so that containers are knows in the network with their names. With docker service its broken.
@rkharya Looking at https://docs.docker.com/engine/reference/commandline/service_create/, I don't see any requirement around container naming. Can you please point us to the requirement you mention above (i.e., the network driver must name containers based of some scheme?)
My quick experiment locally w/o Contiv shows the same behavior as what you see with Contiv above:
➜ ~ docker service create --name redis --replicas=5 redis:3.0.6
0t9yw26srx2tps3xktj6fvuod
Since --detach=false was not specified, tasks will be created in the background.
In a future release, --detach=false will become the default.
➜ ~ docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
0t9yw26srx2t redis replicated 0/5 redis:3.0.6
➜ ~ docker service ps redis
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
0a7w5j1f8h5o redis.1 redis:3.0.6 moby Running Running 11 seconds ago
9jn8v3vvj1tj redis.2 redis:3.0.6 moby Running Running 11 seconds ago
u6hahgnepxkm redis.3 redis:3.0.6 moby Running Running 11 seconds ago
f9mythqsdbpf redis.4 redis:3.0.6 moby Running Running 11 seconds ago
6ig0j6zyrwkj redis.5 redis:3.0.6 moby Running Running 11 seconds ago
➜ ~ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ee2fb01bd10f redis:3.0.6 "/entrypoint.sh re..." 27 seconds ago Up 25 seconds 6379/tcp redis.5.6ig0j6zyrwkjce7thho3zimi9
fb35f8eb5103 redis:3.0.6 "/entrypoint.sh re..." 27 seconds ago Up 25 seconds 6379/tcp redis.1.0a7w5j1f8h5obc8qiaccon8we
7969f8895a38 redis:3.0.6 "/entrypoint.sh re..." 27 seconds ago Up 25 seconds 6379/tcp redis.3.u6hahgnepxkmo7non1n90ackf
59a01c9aa89d redis:3.0.6 "/entrypoint.sh re..." 27 seconds ago Up 25 seconds 6379/tcp redis.2.9jn8v3vvj1tjgub9iryzijmoo
3d217641c73e redis:3.0.6 "/entrypoint.sh re..." 27 seconds ago Up 25 seconds 6379/tcp redis.4.f9mythqsdbpfb8imilrrlzwlf
➜ ~ docker exec -it redis.5.6ig0j6zyrwkjce7thho3zimi9 /bin/sh
# more /etc/hosts
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.17.0.6 ee2fb01bd10f
#
When we don't use docker service and use docker run
and deploy container with Contiv IPAM driver, we do get /etc/hosts appended with container name. And we can reach out to deployed container on the network using that name.
Same is not true with docker service
, which you also saw. So the ask is - as part of Contiv's service discovery feature, will this not be possible to have containers attached to a docker service known on the network through their host names? Essentially, /etc/hosts should get appended with the container names rather than container IDs. Is that falls under the scope of Contiv service discovery feature or its a docker service limitation.
Hope i am able to explain what is intended cause of the issue.
@rkharya, network plugin does not update the /etc/hosts inside the container. This is done by docker in CNM. For Contiv to know the containers attached to the service, Docker should expose the service APIs. Without that the plugins will not know about the service VIP and backend IP mapping. Docker also services the DNS requests and contiv doesn't see it. We need to discuss with docker folks to see how your use case can be addressed
@rkharya https://github.com/docker/libnetwork/pull/1855