[awscontainerinsightreceiver] alternative node metadata source

[bryangardner]

Is your feature request related to a problem? Please describe. In EKS clusters running EC2 nodes with pod access to EC2 IMDS restricted, the awscontainerinsightreceiver fails to fetch instance ID, instance type, instance region, and instance IP address. This causes the receiver to drop all metrics.

Describe the solution you'd like An alternative node metadata source for the awscontainerinsightreceiver, other than IMDS. Some potential sources include:

1. Kubernetes node API
2. EC2 service API

This likely will require the node name to be passed into the pod, documented here.

Describe alternatives you've considered

Additional context

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[bryangardner]

Commenting to remove stale label

[T0tt1]

I am having the same issue. Please work on fixing this bug. I don't think it is a feature but a bug when working with IMDSv2 (Metadata service v2) on EC2. Pods while trying to start just crash and as per the logs of the pod, the issue is: EC2MetadataError: failed to make EC2Metadata request\n\n\tstatus code: 401, request id:

Workaround is to switch EKS Workers' EC2 instance back to IMDSv1 but for me this is not a good and permanent solution. aws ec2 modify-instance-metadata-options --instance-id <instance-id> --http-tokens optional --http-endpoint enabled

Appreciate your assistance in advance.

[bryangardner]

Hey @T0tt1, by default the IMDSv2 response hop limit is 1. Another workaround, if you want to disable IMDSv1 and use IMDSv2, you can try to configure the http-put-response-hop-limit to 2.

From EC2 IMDSv2 docs:

By default, the response to PUT requests has a response hop limit (time to live) of 1 at the IP protocol level. You can adjust the hop limit using the modify-instance-metadata-options command if you need to make it larger. For example, you might need a larger hop limit for backward compatibility with container services running on the instance.

[T0tt1]

Hey @T0tt1, by default the IMDSv2 response hop limit is 1. Another workaround, if you want to disable IMDSv1 and use IMDSv2, you can try to configure the http-put-response-hop-limit to 2.

From EC2 IMDSv2 docs:

By default, the response to PUT requests has a response hop limit (time to live) of 1 at the IP protocol level. You can adjust the hop limit using the modify-instance-metadata-options command if you need to make it larger. For example, you might need a larger hop limit for backward compatibility with container services running on the instance.

@bryangardner do you think this will help? While deploying the ADOT OTEL Namespace, Ds, Svc, Pods, etc. I am getting HTTP 401 in pod logs and they fail to become ready. I will give it a try.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions[bot]]

This issue was closed because it has been marked as stale for 30 days with no activity.

[hammopauGSS]

Ignoring the problem does not make it go away AWS...