Show "real" Pods running behind a service

[edouardKaiser]

Issue details

The Service details page is listing the pods behind a Service. It seems like it's displaying the pods matching the selector. Problem is, it looks like the pods are being served by the service, which is not necessarily true.

Example, the Service Details shows 2 Pods, but in reality, my service is only serving one pod (because they other one is starting/not healthy/shutting down/etc...):

➜  ~ kubectl describe svc/booking-engine-ph-svc --namespace="staging"
Name:           booking-engine-ph-svc
Namespace:      staging
Labels:         app=booking-engine-ph
Selector:       app=booking-engine-ph
Type:           NodePort
IP:         10.3.0.233
Port:           <unset> 80/TCP
NodePort:       <unset> 31606/TCP
Endpoints:      10.2.6.4:9000
Session Affinity:   None
No events.

Environment

Dashboard version: 1.4.1
Kubernetes version: 1.4
Operating system: CoreOS

It's nice to see all the Pods matching the selector, but it would also be great to know which one are really being served (new flag? icon for non healthy/not served ?)

[floreks]

Does kubectl get/describe for this pod shows some additional info, i.e. that this other pod (container in pod) is failing/crashing?

The reason why I'm asking is that we want to mimic kubectl best we can and if it says that everything is fine then we would show the same.

[edouardKaiser]

If you run a kubectl describe pods/<name> you get some information on the status:

You can see in the "Conditions" section, it says Ready: false :

Conditions:
  Type      Status
  Initialized   True 
  Ready     False 
  PodScheduled  True 
Volumes:
  configuration-volume:
    Type:   ConfigMap (a volume populated by a ConfigMap)
.....

But even if you don't want to go down the path of displaying information about the pod's status. I think it would be good to reflect on the service details page, one way or another, the endpoints served by the service (information that you get from kubectl describe service).

[ianlewis]

Some things to note. These are mostly observations/suggestions.

1. If we just filter the pods that aren't ready would that suffice? Are there other edge cases where pods that match the selector are excluded from the endpoints?
2. It almost would make more sense to show info about the "endpoints" with service/endpoint specific info (like the pod IP) rather than the "pods". i.e. list the "Endpoints" here. IP address, and some info about. Maybe the dashboard can use the endpoints API internally to get the list rather than re-implementing all the logic used to get the endpoints?
3. We don't show the ready state of the pods in the list view. If we showed ready state would that solve the problem?

[bryk]

We don't show the ready state of the pods in the list view. If we showed ready state would that solve the problem?

@edouardKaiser Would it help if we shown the state column here? And allow for filtering/sorting on it?

[edouardKaiser]

I think it would reflect the reality 99.9% of the time. But the real source of truth is the kubect describe svc/<yourservice> which returns the endpoints being served.

[rf232]

I think it would also help us if we get the state icons more nicely done (#1349) and use them here. Since IMHO It would be nice to see that (in the example screenshot) this 2nd pod is expected to join serving for this service. Especially if there are a 'lot' of pods behind the service reading a column takes more time than scanning for differently colored icons in the list

[bryk]

I think it would reflect the reality 99.9% of the time. But the real source of truth is the kubect describe svc/ which returns the endpoints being served.

It is also shown on the service page, so I think we're aligned here.

[edouardKaiser]

You mean the endpoints that the service is serving are displayed on the service details page?

[bryk]

You mean the endpoints that the service is serving are displayed on the service details page?

Yes.

[edouardKaiser]

But the endpoints are not displayed right now, unless I missed something.

[ianlewis]

I think it makes sense to show all the pods including those that aren't ready but show those with a different icon and state per #1349.

Just showing the pods that match the endpoints would leave off important info about pods that match the selector but aren't ready for various reasons. For that reason, I think that showing the pods that match the endpoints is not what we want but the pods that match the selector + ready state.

Though, do have to think a bit about services that have no selector (headless) but have manually updated endpoints? Is that a thing?

[bryk]

But the endpoints are not displayed right now, unless I missed something.

They are:

[bryk]

For that reason, I think that showing the pods that match the endpoints is not what we want but the pods that match the selector + ready state.

Yes, thats the correct behavior, IMO.

Though, do have to think a bit about services that have no selector (headless) but have manually updated endpoints? Is that a thing?

We should (and we do) show them. Then we should surface all information, potentially about the configuration that made the headless service work.

[edouardKaiser]

I'm not sure what are those endpoinds, but it does not really match that we get from kubectl describe svc/<service name>.

Example of console output, we can see 2 IP addresses being served, 10.2.6.2 and 10.2.73.3:

Name:           booking-engine-au-svc
Namespace:      production
Labels:         app=booking-engine-au
Selector:       app=booking-engine-au
Type:           NodePort
IP:         10.3.0.139
Port:           <unset> 80/TCP
NodePort:       <unset> 30905/TCP
Endpoints:      10.2.6.2:9000,10.2.73.3:9000
Session Affinity:   None
No events.

UI of this service, does not display 10.2.6.2 and 10.2.73.3 in the endpoints section:

[floreks]

UI of this service, does not display 10.2.6.2 and 10.2.73.3 in the endpoints section

That's true. We are displaying endpoints of a service but not endpoints of a pods targeted by this service. Usually user will not have access to these "pod endpoints". Normally you would have to be inside a cluster to access them.

We can display this information, but I'm not sure if it will be useful. Do you have any use cases for that in mind?

[edouardKaiser]

I find it useful because it's the source of truth of which pods are being served by the service.

[floreks]

This endpoint is additional kubernetes resource, so maybe we could show them as a related object and use some of that logic to match pods to a service better.

$ kubectl describe endpoints heapster --namespace kube-system
Name:       heapster
Namespace:  kube-system
Labels:     app=heapster
        kubernetes.io/cluster-service=true
        kubernetes.io/name=Heapster
Subsets:
  Addresses:        10.1.20.12
  NotReadyAddresses:    <none>
  Ports:
    Name    Port    Protocol
    ----    ----    --------
    <unset> 8082    TCP

No events.

$ kubectl get endpoints --namespace kube-system
NAME                      ENDPOINTS                       AGE
heapster                  10.1.20.12:8082                 23h
influxdb                  10.1.20.4:8086,10.1.20.4:8083   23h
kube-dns                  10.1.20.3:53,10.1.20.3:53       23h
kubernetes-dashboard      10.1.20.2:9090                  23h

What do you think?

[edouardKaiser]

Interesting, I didn't know they were proper resources. As long as we can access/display those values, I'm happy with it.

[bryk]

This endpoint is additional kubernetes resource, so maybe we could show them as a related object and use some of that logic to match pods to a service better.

I like this idea. Is this something you can work on? :)

[ianlewis]

Yah. "internal endpoints" is different. I think showing endpoints, one way or the other is good but they will almost always be pods so that's the main use case, but pods ≈ endpoints.

The trick is showing this stuff in a way that's useful.

On Mon, Oct 24, 2016, 22:03 Edouard Kaiser [email protected] wrote:

I'm not sure what are those endpoinds, but it does not really match that we get from kubectl describe svc/.

Example of console output, we can see 2 IP addresses being served, 10.2.6.2 and 10.2.73.3:

Name: booking-engine-au-svc Namespace: production Labels: app=booking-engine-au Selector: app=booking-engine-au Type: NodePort IP: 10.3.0.139 Port: 80/TCP NodePort: 30905/TCP Endpoints: 10.2.6.2:9000,10.2.73.3:9000 Session Affinity: None No events.

UI of this service, does not display 10.2.6.2 and 10.2.73.3 in the endpoints section: [image: image] https://cloud.githubusercontent.com/assets/7731641/19646685/50d0e22e-9a46-11e6-9b38-f78cca410579.png

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kubernetes/dashboard/issues/1317#issuecomment-255734131, or mute the thread https://github.com/notifications/unsubscribe-auth/AADAiTvEBRYZfrz4AcMpDIi6qOBejCR7ks5q3KyqgaJpZM4KSKTD .

[floreks]

I like this idea. Is this something you can work on? :)

Probably yes but it can take some time as I have some other open issues in core :)

[gauravarora]

This would be very useful from a monitoring perspective also. For example, if I expect 4 pods in my deployment but only see 3 endpoints in the service then I know I have a problem with the pod.

[djschny]

Listing all pods behind a service (label matching) and then column for wether each one is in service or not would be immensely helpful. It shows at a glance which ones have failed a readiness probe for example.

[floreks]

@djschny I'm not sure if I understand correctly. Currently on service detail page you can see list of pods targeting chosen service. They are matched using labels. I think that this is how kubernetes matches them also in order to route the traffic.

column for wether each one is in service or not would be immensely helpful.

What do you mean by that exactly?

[djschny]

@floreks what I mean is there is a distinction between what pods are supposed to be targeted by the service vs. which ones are actively being targeted. If a pod has the correct matching labels but the readiness probe is failing, then the pod will not receive traffic.

Think of the status page for HAProxy where it lists all hosts that are part of the backend pool and then shows colors red, green, yellow accordingly for hosts based upon whether traffic is going to them.

[floreks]

~Readiness~Liveness probe has to be defined in resource definition during creation. If it is defined then kubernetes is responsible for checking if application is healthy. In case this check fails pod is restarted. We probably would be able to extract this data also but if pod is constantly being restarted then you can see that on details page just by looking at status icon. I'm not sure what else we could do here.

Sample part in yaml looks like that:

livenessProbe:
  httpGet:
    scheme: HTTPS
    path: /
    port: 8443
  initialDelaySeconds: 30
  timeoutSeconds: 30

[djschny]

I believe readinessProbe is being mistaken/confused with livenessProbe. I’m referring to readinessProb which does not affect the restarting of a Pod, but rather whether traffic is routed to it by the Service.

[floreks]

Ahh... you are right. My mistake. So you are defining readinessProbe for your resources right? We could use that to differentiate between actively targeted pods and not.

[djschny]

Yep, exactly. Sorry if I was not clear enough in the beginning of communication of this issue.

So if we step back to the original request in this issue, I was adding a +1 to adding additional fields/context showing which one of the matching pods are actively having traffic being routed to them.

Thanks for going back and forth with me on this.

[floreks]

Good that we got to conclusion here :) We will take a look at this. Should not be that hard to implement.