kube-netc icon indicating copy to clipboard operation
kube-netc copied to clipboard

Metrics incorrectly being reported for pods that don't have any ports

Open patelrit opened this issue 4 years ago • 0 comments

I installed kube-netc in on of our clusters and checked the metrics. I noticed that bytes_recv being reported for pods that don't have any ports.

e.g. nirmata-cni-installer-5fkmp pod is part of a daemonset and does not have any ports configured. Also 10.10.1.210:2379 is actually etcd container running on the node (outside kubernetes)

Also, another observation is that there are multiple records between the same source and destination IP. This increases the size of the metrics data creating scale issues with prometheus.

bytes_recv{destination_address="10.10.1.210:2379",destination_pod_name="nirmata-cni-installer-5fkmp",source_address="10.10.1.210:49080",source_pod_name="nirmata-cni-installer-5fkmp"} 459 bytes_recv{destination_address="10.10.1.210:2379",destination_pod_name="nirmata-cni-installer-5fkmp",source_address="10.10.1.210:49082",source_pod_name="nirmata-cni-installer-5fkmp"} 1.791227e+06 bytes_recv{destination_address="10.10.1.210:2379",destination_pod_name="nirmata-cni-installer-5fkmp",source_address="10.10.1.210:49084",source_pod_name="nirmata-cni-installer-5fkmp"} 787 bytes_recv{destination_address="10.10.1.210:2379",destination_pod_name="nirmata-cni-installer-5fkmp",source_address="10.10.1.210:49090",source_pod_name="nirmata-cni-installer-5fkmp"} 29955 bytes_recv{destination_address="10.10.1.210:2379",destination_pod_name="nirmata-cni-installer-5fkmp",source_address="10.10.1.210:49092",source_pod_name="nirmata-cni-installer-5fkmp"} 2026

DaemonSet spec (partial) spec: containers: - image: index.docker.io/nirmata/nirmata-cni-installer:1.10 imagePullPolicy: IfNotPresent livenessProbe: exec: command: - cat - /run.sh failureThreshold: 3 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 name: install-cni readinessProbe: exec: command: - cat - /run.sh failureThreshold: 3 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 resources: limits: memory: 200Mi requests: cpu: 100m memory: 100Mi securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true runAsGroup: 1000 runAsNonRoot: true runAsUser: 1000 terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /opt/cni/bin/ name: cni-bin dnsPolicy: ClusterFirst hostNetwork: true imagePullSecrets: - name: default-registry-secret initContainers: - command: - chown - -R - 1000:1000 - /opt/cni/bin/ image: alpine:3.6 imagePullPolicy: IfNotPresent name: take-data-dir-ownership resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /opt/cni/bin/ name: cni-bin restartPolicy: Always schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 30 tolerations: - effect: NoSchedule key: node-role.kubernetes.io/master operator: Exists volumes: - hostPath: path: /opt/cni/bin type: "" name: cni-bin updateStrategy: type: OnDelete

patelrit avatar Jun 21 '20 22:06 patelrit