Intermittent error: "CredentialsProviderError: Could not load credentials from any providers"

[dmacbale-cas]

I believe I'm having a similar issue to https://github.com/awslabs/amazon-ecs-local-container-endpoints/issues/89 but it's different enough that I'm opening a new issue. (Somewhat similar to https://github.com/awslabs/amazon-ecs-local-container-endpoints/issues/118 and maybe https://github.com/awslabs/amazon-ecs-local-container-endpoints/issues/35)

As snehal-kolte mentioned, I'm following the same document as testing containers: https://aws.amazon.com/blogs/compute/a-guide-to-locally-testing-containers-with-amazon-ecs-local-endpoints-and-docker-compose/

And also referencing the docs for how to vend credentials that are in this repo: https://github.com/awslabs/amazon-ecs-local-container-endpoints/blob/mainline/docs/features.md#vend-credentials-to-containers

Most of the time there are no issues but on occasion the following error appears in the logs:

{"level":"error","message":"Unhandled promise rejection: CredentialsProviderError: Could not load credentials from any providers","meta":{},"timestamp":"2022-03-22T21:26:36.174Z"}
You have triggered an unhandledRejection, you may have forgotten to catch a Promise rejection:
CredentialsProviderError: Could not load credentials from any providers
    at providers (/<project-dir>/<project-dir>/node_modules/@aws-sdk/credential-provider-node/dist-cjs/index.js:25:19)
    at /<project-dir>/<project-dir>/node_modules/@aws-sdk/property-provider/dist-cjs/chain.js:11:28
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async coalesceProvider (/<project-dir>/<project-dir>/node_modules/@aws-sdk/property-provider/dist-cjs/memoize.js:13:24)
    at async SignatureV4.credentialProvider (/<project-dir>/<project-dir>/node_modules/@aws-sdk/property-provider/dist-cjs/memoize.js:32:24)
    at async SignatureV4.signRequest (/<project-dir>/<project-dir>/node_modules/@aws-sdk/signature-v4/dist-cjs/SignatureV4.js:84:29)
    at async /<project-dir>/<project-dir>/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:13:18
    at async StandardRetryStrategy.retry (/<project-dir>/<project-dir>/node_modules/@aws-sdk/middleware-retry/dist-cjs/StandardRetryStrategy.js:51:46)
    at async /<project-dir>/<project-dir>/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:6:22

The workaround has been running the same command again to hit our endpoint and it will always work (we never see the error back-to-back).

My docker-compose.override.yml:

version: "2"
networks:
    credentials_network:
        driver: bridge
        ipam:
            config:
                - subnet: "169.254.170.0/24"
                  gateway: 169.254.170.1
services:
    ecs-local-endpoints:
        image: amazon/amazon-ecs-local-container-endpoints
        volumes:
            - /var/run:/var/run
            - $HOME/.aws/:/home/.aws/
        environment:
            AWS_PROFILE: "$AWS_PROFILE"
            HOME: "/home"
        networks:
            credentials_network:
                ipv4_address: "169.254.170.2"

    app:
        depends_on:
            - ecs-local-endpoints
        networks:
            credentials_network:
                ipv4_address: "169.254.170.3"
        environment:
            AWS_DEFAULT_REGION: "us-east-1"
            AWS_CONTAINER_CREDENTIALS_RELATIVE_URI: "/role/$APP_AWS_TASK_ROLE"
            ECS_CONTAINER_METADATA_URI: "http://169.254.170.2"
            ECS_CONTAINER_METADATA_URI_V4: "http://169.254.170.2/v3"

• I don't seem to be able to consistently duplicate this error because the credentials work most of the time. But pretty regularly I see the above error.
• I'm running 1.4.0 of the amazon-ecs-local-container-endpoints.
• No modifications have been made.
• Nothing unusual or noteworthy of my environment or deployment.