Sourcecategories in Sumo appending numbers

[jeffwroblewski]

For 2.1 and beyond, user is seeing an issue where sourecategories are populating like this:

app_prod/some_app_name app_prod/some_app_name_62 app_prod/some_app_name 63...

Can provide more info offline as needed.

Thanks! Jeff W. TAM, Sumo

[frankreno]

cc @bendrucker : ill look into this as soon as I can, but seems likely related to the fix for #78

[bendrucker]

What version of Kubernetes are you running?

[andrews32]

We're running OpenShift v3.3 which includes Kubernetes v1.3.

[bendrucker]

Gotcha, seems like there's probably no test coverage for that pod name format anymore. I can look into it in a few.

[andrews32]

We are in the middle of upgrading to OpenShift v3.9 which includes Kubernetes v1.9, but the symbolic links where I believe the code retrieves the pod name from is the same format as in K8S v1.3.

For example, the first one is docker-registry-2-mqe0f... Where docker-registry is the pod_name, 2 is the deployment config counter, and mqe0f is the hash.

The problem is its inconsistent on what is retrieves as the _sourceCategory. Sometimes it'll be "docker-registry", sometimes it'll be "docker-registry-2".

[root@infra01-devtest-vxbyr ~]# ls /var/log/containers/ docker-registry-2-mqe0f_default_POD-9171d6915e911a532fb6048191e9713ed36a14ccd1a9057624ece298f08b350a.log docker-registry-2-mqe0f_default_registry-6c077b90f9592770d1e63b0444551d331167551e035f3d25b5922d0b4ec05325.log hawkular-cassandra-1-bd5m3_openshift-infra_hawkular-cassandra-1-86f7ee9fddbff12f935a88b0b54e7b46e82a73657000ff41b209076ed7fcc657.log hawkular-cassandra-1-bd5m3_openshift-infra_POD-a6d27b85d71c4e0c56e600b8d3666e39da8d360515c75d20282318b56f50be47.log hawkular-metrics-5ovsm_openshift-infra_hawkular-metrics-e70c33b6df41717ad12ccfc1d55b462603ac6a79adbae45c7cad0d363bfccd74.log hawkular-metrics-5ovsm_openshift-infra_POD-02596538d07c076a7c2447bd364981296ed2584aea29e1158eca643d78359953.log registry-console-1-6iu26_default_POD-7ce336ef1bb0442e3d93c642c7b63c523e23adf924e4ed1f4f26bd7db6e17c64.log registry-console-1-6iu26_default_registry-console-a3b0ad6a98b33001ef205bd1bb83d027d19d67013f701d66ec05183314a04e3c.log router-25-2gpi9_default_POD-ec02bb74c544cc201fea714ce52b3f7bef9adb7d6c47b03e7b63df4cb8df6819.log router-25-2gpi9_default_router-6cea4aef80d33705c275d19392de0608accec18243e7ef2a7a773103735ee510.log

[bendrucker]

So the actual pod name is docker-registry-2-mqe0f if I'm reading right? Would be a huge help if you could get an entire pod (kubectl get pod <name> -o yaml) for confirmation.

[bendrucker]

I suspect the regression that's affecting you from #78 has to do with the pod template hash. Rather than hardcode error-prone patterns based on string formatting (i.e. strip this part if it's numbers), we switched to actually detecting the pod template hash and deterministically stripping the dynamic parts. I'm trying to get a 1.3 cluster up on minikube but in case that doesn't seem viable so a full pod from your cluster would be helpful.

[andrews32]

`[svc-vxby-ose@master01-devtest-vxbyr ~]$ oc get po docker-registry-3-9n3bx -o yaml apiVersion: v1 kind: Pod metadata: annotations: kubernetes.io/created-by: | {"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicationController","namespace":"default","name":"docker-registry-3","uid":"3bf3e47f-fad7-11e8-8df7-005056848c95","apiVersion":"v1","resourceVersion":"1205030190"}} openshift.io/deployment-config.latest-version: "3" openshift.io/deployment-config.name: docker-registry openshift.io/deployment.name: docker-registry-3 openshift.io/scc: restricted creationTimestamp: 2018-12-08T10:52:07Z generateName: docker-registry-3- labels: deployment: docker-registry-3 deploymentconfig: docker-registry docker-registry: default name: docker-registry-3-9n3bx namespace: default resourceVersion: "1205031022" selfLink: /api/v1/namespaces/default/pods/docker-registry-3-9n3bx uid: 4e5362ec-fad7-11e8-8df7-005056848c95 spec: containers:

• env:
  • name: REGISTRY_HTTP_ADDR value: :5000
  • name: REGISTRY_HTTP_NET value: tcp
  • name: REGISTRY_HTTP_SECRET value: vN4FVfWmHghp7shKhjZadrA6HLg+9FAPqEORak7+VFQ=
  • name: REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ENFORCEQUOTA value: "false"
  • name: REGISTRY_HTTP_TLS_CERTIFICATE value: /etc/secrets/registry.crt
  • name: REGISTRY_HTTP_TLS_KEY value: /etc/secrets/registry.key image: openshift3/ose-docker-registry:v3.3.1.46.45 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 3 httpGet: path: /healthz port: 5000 scheme: HTTPS initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 5 name: registry ports:
  • containerPort: 5000 protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 5000 scheme: HTTPS periodSeconds: 10 successThreshold: 1 timeoutSeconds: 5 resources: requests: cpu: 100m memory: 256Mi securityContext: capabilities: drop:
    • KILL
    • MKNOD
    • SETGID
    • SETUID
    • SYS_CHROOT privileged: false runAsUser: 1000000000 seLinuxOptions: level: s0:c1,c0 terminationMessagePath: /dev/termination-log volumeMounts:
  • mountPath: /registry name: registry-storage
  • mountPath: /etc/secrets name: volume-zx4oi
  • mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: registry-token-sbvmg readOnly: true dnsPolicy: ClusterFirst host: infra04-devtest-vxbyr.xxxx.com imagePullSecrets:
• name: registry-dockercfg-8xfow nodeName: infra04-devtest-vxbyr.xxxx.com nodeSelector: region: infra restartPolicy: Always securityContext: fsGroup: 1000000000 seLinuxOptions: level: s0:c1,c0 serviceAccount: registry serviceAccountName: registry terminationGracePeriodSeconds: 30 volumes:
• emptyDir: {} name: registry-storage
• name: volume-zx4oi secret: secretName: registry-certificates
• name: registry-token-sbvmg secret: secretName: registry-token-sbvmg status: conditions:
• lastProbeTime: null lastTransitionTime: 2018-12-08T10:52:07Z status: "True" type: Initialized
• lastProbeTime: null lastTransitionTime: 2018-12-08T10:52:37Z status: "True" type: Ready
• lastProbeTime: null lastTransitionTime: 2018-12-08T10:52:07Z status: "True" type: PodScheduled containerStatuses:
• containerID: docker://a1ce41ffcfcebd69e7f8887493db8e0e7636467862e62a63f9c6823f996fef2a image: openshift3/ose-docker-registry:v3.3.1.46.45 imageID: docker-pullable://registry.access.redhat.com/openshift3/ose-docker-registry@sha256:7b429aa43daf2a2d63c968f685a1c42481055fb14dd68678467f8d0de94d89eb lastState: {} name: registry ready: true restartCount: 0 state: running: startedAt: 2018-12-08T10:52:28Z hostIP: 10.224.210.31 phase: Running podIP: 10.221.76.3 startTime: 2018-12-08T10:52:07Z `

[bendrucker]

From what you posted, the deployment name is docker-registry-3. This repo was meant to remove random sections included from Deployments/ReplicaSets, not necessarily any numeric ID. Seems like it was a bug that it matched/deleted part of your deployment name from the pod_name. You could consider using the open shift labels directly for your source categories.

[andrews32]

I'm guessing deployment name wasn't always where the _sourceCategory got his values from. This is new behavior.

Also, as Frank mentiond above, #78 was only fixed/closed in December 2018 which matches the first reports of this new behavior.

What changed in #78 and why? How do we undo it without manually using an old version that will not be maintained?

[bendrucker]

I'm guessing deployment name wasn't always where the _sourceCategory got his values from.

I don't see any reason to assume that

#78 was closed by #100. #78 identified bugs in the original naive implementation of replica pod sanitization. The original implementation would remove the second to last segment of the pod name if it were a number. This is unnecessarily naive.

This numeric value was the pod template hash which is included as a label on the pods. In later versions of k8s, that numeric value was mapped to an alphanumeric encoding, breaking the naive name sanitization. #100 takes the template hash, looks for the numeric or alphanumeric version in the pod name, and removes that segment by exact match. Anything else you do to your pods, including numbers, is left behind.

This feature was meant to target Kubernetes ReplicaSets and this plugin was stripping bits of your pod name due to a bug. It sucks, but sometimes bug fixes are breaking changes if you were depending on buggy behavior.

I made some suggestions above on how to provide a specific metadata template with labels—that would let you define conventions that match your stack. I don't think it would be a good idea to re-introduce behavior that parses pod name conventions outside of what's present in k8s core.

[andrews32]

I was looking for something else and came across this ticket and noticed it was still open. I think we can close this now. Thanks for explaining it @bendrucker.