Missing endpoint details with additionalPortAnnotations is used

[dinesh]

I'm using additionalPortAnnotations to discover the pods where the port is not explicitly mentioned in ports but annotated. But it seems not all details are populated in the endpoints like container_image, container_id etc.

For example -

in helm values -

additionalPortAnnotations: 
- prometheus.io/port

$ signalfx-agent status endpoints

 ...
  - internalId: teleport-proxyauth-5585f6964c-vx5b6-0db94de-3000
   alternate_port: 0
   container_command: 
   container_id: 
   container_image: 
   container_labels: map[]
   container_names: []
   container_spec_name: teleport
   container_state: 
   discovered_by: k8s-api
   has_port: true
   host: 10.122.109.102
   id: teleport-proxyauth-5585f6964c-vx5b6-0db94de-3000
   ip_address: 10.122.109.102
   kubernetes_annotations: map[checksum/config:003f9d954b80f054a9c46d34270760374ad74b406d09fdb49f0f973aa6057408 kubectl.kubernetes.io/restartedAt:2021-04-21T15:20:01Z kubernetes.io/psp:eks.privileged prometheus.io/port:3000]
   kubernetes_namespace: teleport-proxy
   kubernetes_pod_name: teleport-proxyauth-5585f6964c-vx5b6
   kubernetes_pod_uid: 0db94de0-23bb-4b71-89e2-24f5320915b8
   name: 
   network_port: 3000
   orchestrator: 1
   port: 3000
   port_labels: map[]
   port_type: UNKNOWN
   private_port: 3000
   public_port: 0
   target: hostport

I want to add the discovery rule by container_image or container_labels.

[keitwb]

When you do kubectl get pod -o yaml -n teleport-proxy teleport-proxyauth-5585f6964c-vx5b6 does the resulting yaml show the container image and labels under the container spec? If so, this might be a bug in the k8s-api observer. Otherwise that is the only place the observer has to get the information from so it would be more an issue with what K8s is reporting.

[dinesh]

It does have the labels and image set (I cross-checked again).

[keitwb]

Ok, @dinesh I looked a bit more into this and I understand why it is happening.

When you specify the port via annotations, it doesn't know which container the port is coming from since all containers share the same networking namespace. That is why you won't get container image info. In the case of a single container in a pod, we could just pass through that container image, but it becomes trickier to decide what the container image should be with multiple containers since there is no way to identify which container the port comes from (if it were in the port spec, there wouldn't be any real need to use the port annotations).

However, you can get pod label information from the pod_metadata field in newer versions of the agent. Labels are under pod_metadata.labels, which would be available even when using port annotations. It looks like your agent might be an older version that doesn't have it so you might have to upgrade.

[atoulme]

Closing this issue as inactive. Please reopen if this issue is still occurring.