Unable to access pods directly via the pod IPs when the pods are exposed as a service

[alexbrand]

Description

I configured contiv with K8s, and ran our smoke test utility (https://github.com/apprenda/kuberang) which performs a couple of networking checks.

This tool deploys an nginx pod and a busybox pod, exposes the nginx pod as a service, and ensures that the busybox pod is able to wget the nginx pod directly via the pod IP, the service IP and DNS.

When running against a cluster with Contiv, the pod IP check fails because wget times out trying to reach the nginx pod directly via the pod IP.

Expected Behavior

The busybox pod should be able to connect to the nginx pod using the pod IP.

Observed Behavior

The busybox pod times out trying to connect to the nginx pod using the pod IP.

Steps to Reproduce (for bugs)

Feel free to run https://github.com/apprenda/kuberang against a kubernetes cluster.

Your Environment

• netctl version 1.1.1
• Orchestrator version (e.g. kubernetes, mesos, swarm): Kubernetes v1.7.0
• Operating System and version Ubuntu 16.04 and CentOS 7

~# netctl global info
Fabric mode: default
Forward mode: routing
ARP mode: proxy
Vlan Range: 1-4094
Vxlan range: 1-10000
Private subnet: 172.19.0.0/16

~# netctl net ls
Tenant   Network      Nw Type  Encap type  Packet tag  Subnet         Gateway     IPv6Subnet  IPv6Gateway  Cfgd Tag
------   -------      -------  ----------  ----------  -------        ------      ----------  -----------  ---------
default  contivh1     infra    vxlan       0           132.1.1.0/24   132.1.1.1
default  default-net  data     vxlan       0           172.16.0.0/16  172.16.0.1

[alexbrand]

I have noticed that this behavior changes depending on what is accessed first.

The pod IPs and service IPs are accessible if the pod IPs are accessed before the service IP is initially accessed.

The pod IPs are unreachable if the first request issued by the busybox pod is to the service IP.

[rchirakk]

@alexbrand that's right We don't allow pod IP Address access after service ip is used. do you guys have some use-case that requires access to pod ip address (for services)? pod ip will be accessible from hosts for health check. And you can access it from busybox using service/cluster-ip or service-name.