amazon-cloudwatch-agent icon indicating copy to clipboard operation
amazon-cloudwatch-agent copied to clipboard

cloudwatch-agent/cloudwatch-agent:1.300028.1b210 breaks Fargate integration

Open jeffrigby opened this issue 1 year ago • 4 comments

The latest image (cloudwatch-agent/cloudwatch-agent:1.300028.1b210) broke my previously working Fargate instance. I include it in my SAM template task definition.

        - Name: cloudwatch-agent
          Image: "public.ecr.aws/cloudwatch-agent/cloudwatch-agent:latest"
          Secrets:
            - Name: CW_CONFIG_CONTENT
              ValueFrom: !Ref CWAgentConfigSecret
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-group: !Ref StatsLogGroup
              awslogs-region: !Ref AWS::Region
              awslogs-stream-prefix: ecs
          Essential: true
          Environment:
            - Name: AWS_EMF_ENVIRONMENT
              Value: ECS
            - Name: AWS_EMF_SERVICE_TYPE
              Value: ECS
            - Name: AWS_EMF_LOG_GROUP_NAME
              Value: !Ref StatsLogGroup
            - Name: AWS_EMF_NAMESPACE
              Value: "App/Name"

My secret is just the default:

{"agent":{"log_level":"INFO"},"logs":{"metrics_collected":{"emf":{}}}}

It worked fine with the previous version (1.300026.3b189), but the new version can't send the data. Reverting and fixing the previous version (not latest) resolved the issues.

I get these logs:

September 27, 2023 at 17:09 (UTC-4:00)	2023/09/27 21:09:22 D! should retry true for imds error : RequestError: send request failed	cloudwatch-agent
September 27, 2023 at 17:09 (UTC-4:00)	caused by: Put "http://169.254.169.254/latest/api/token": dial tcp 169.254.169.254:80: connect: invalid argument	cloudwatch-agent
September 27, 2023 at 17:09 (UTC-4:00)	2023/09/27 21:09:22 D! could not get instance document without imds v1 fallback enable thus enable fallback	cloudwatch-agent
September 27, 2023 at 17:09 (UTC-4:00)	E! [EC2] Fetch hostname from EC2 metadata fail: RequestError: send request failed	cloudwatch-agent
September 27, 2023 at 17:09 (UTC-4:00)	caused by: Get "http://169.254.169.254/latest/meta-data/hostname": dial tcp 169.254.169.254:80: connect: invalid argument	cloudwatch-agent
September 27, 2023 at 17:09 (UTC-4:00)	2023/09/27 21:09:22 D! should retry true for imds error : RequestError: send request failed	cloudwatch-agent
September 27, 2023 at 17:09 (UTC-4:00)	caused by: Put "http://169.254.169.254/latest/api/token": dial tcp 169.254.169.254:80: connect: invalid argument	cloudwatch-agent
September 27, 2023 at 17:09 (UTC-4:00)	2023/09/27 21:09:21 D! should retry true for imds error : RequestError: send request failed	cloudwatch-agent
September 27, 2023 at 17:09 (UTC-4:00)	caused by: Put "http://169.254.169.254/latest/api/token": dial tcp 169.254.169.254:80: connect: invalid argument	cloudwatch-agent
September 27, 2023 at 17:09 (UTC-4:00)	2023/09/27 21:09:21 D! could not get hostname without imds v1 fallback enable thus enable fallback	cloudwatch-agent
September 27, 2023 at 17:09 (UTC-4:00)	D! [EC2] Found active network interface	cloudwatch-agent
September 27, 2023 at 17:09 (UTC-4:00)	2023/09/27 21:09:21 I! imds retry client will retry 1 times	cloudwatch-agent
September 27, 2023 at 17:09 (UTC-4:00)	2023/09/27 21:09:21 D! should retry true for imds error : RequestError: send request failed	cloudwatch-agent
September 27, 2023 at 17:09 (UTC-4:00)	caused by: Put "http://169.254.169.254/latest/api/token": dial tcp 169.254.169.254:80: connect: invalid argument	cloudwatch-agent

How can I get it to fetch a token?

jeffrigby avatar Sep 27 '23 22:09 jeffrigby