Do not collect pod metrics using Bottlerocket

[SpringMT]

Describe the bug When using Bottlerocket in EKS, otel-collector can not collect Pod metrics. Log is bellow.

2022-03-20T14:04:42.986Z	warn	cadvisor/container_info_processor.go:93	No pod metric collected	{"kind": "receiver", "name": "awscontainerinsightreceiver", "metrics count": 27}

Steps to reproduce

1. Create cluster with Bottlerocket
2. Set aws-otel-collector with aws-otel-collector/deployment-template/eks/otel-container-insights-infra.yaml

• I use amazon/aws-otel-collector:v0.17.0 image

What did you expect to see? This bug is similar to this issue https://github.com/aws/amazon-cloudwatch-agent/issues/188. This issue fixed by https://github.com/aws/amazon-cloudwatch-agent/pull/189/.

aws-otel-collector uses container_info_processor in opentelemetry-collector-contrib.

https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/awscontainerinsightreceiver/internal/cadvisor/container_info_processor.go

I think the similar changes in https://github.com/aws/amazon-cloudwatch-agent/pull/189/ will fix this bug.

What did you see instead?

Environment Bottlerocket in EKS

Additional context

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[mhausenblas]

@SpringMT can you please confirm that this still happens with v0.18 of the ADOT collector?

[YDKK]

@mhausenblas I have same issue and have confirmed that this still happen with the amazon/aws-otel-collector:v0.18.0 image.

[YDKK]

As a workaround, I created a VolumeMount from /run/dockershim.sock to /run/containerd/containerd.sock (the same way as CloudWatch Agent) and it works fine.

[vasireddy99]

Adding context here:

Interesting that bottle rocket has fixed the mount point to containerd for kubernetes-1.23 support. so yes, if using latest version of bottle rocket mount needs to be fixed to /run/containerd/containerd.sock.

We are currently working on this issue and shall provide more context as soon as we have an update on resolving.

[vasireddy99]

Thank you so much for creating this issue and appreciate for the good detail. The Support to collect pod metrics while running on EKS cluster with containerd runtime is available from ADOT collector v0.21. Closing this issue.

[SpringMT]

Thank you!