Unable to STS assume Role in fluentbit/S3 plugin

[sabdalla80]

I am am unable to assume a role all of a sudden from my daemonset/EKS application. The fluentbit application is trying to assume a role in a different account so it can write the logs to a bucket there. I am seeing this error recently without knowing what changed to cause this error. I appreciate any feedback on this.

My output:

[OUTPUT] Name s3 Match internal.* region us-east-2 bucket customer-logs-bucket-us-east-2 external_id someId role_arn arn:aws:iam::account2:role/roletoassume sts_endpoint https://sts.us-east-2.amazonaws.com store_dir /tmp/fluent-bit/s3-kube retry_limit 10 total_file_size 15M upload_timeout 15s store_dir_limit_size 50M s3_key_format $TAG-$UUID s3_key_format_tag_delimiters ._ compression gzip

The error from logs with debug on:

[2024/05/26 19:18:24] [ info] [filter:kubernetes:kubernetes.3] connectivity OK [2024/05/26 19:18:24] [ info] [input:emitter:tag_for_s3] initializing [2024/05/26 19:18:24] [ info] [input:emitter:tag_for_s3] storage_strategy='memory' (memory only) [2024/05/26 19:18:24] [ info] [input:emitter:tag_for_kube] initializing [2024/05/26 19:18:24] [ info] [input:emitter:tag_for_kube] storage_strategy='memory' (memory only) [2024/05/26 19:18:24] [ info] [fstore] created root path /tmp/fluent-bit/s3-kube/customer-logs-bucket-us-east-2 [2024/05/26 19:18:24] [ info] [output:s3:s3.0] Using upload size 15000000 bytes [2024/05/26 19:18:24] [ info] [aws_client] auth error, refreshing creds [2024/05/26 19:18:24] [error] [aws_credentials] Shared credentials file /root/.aws/credentials does not exist [2024/05/26 19:18:24] [ info] [output:s3:s3.0] worker #0 started [2024/05/26 19:18:24] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020

[2024/05/26 19:19:42] [ info] [aws_client] auth error, refreshing creds [2024/05/26 19:19:42] [error] [aws_credentials] Shared credentials file /root/.aws/credentials does not exist [2024/05/26 19:19:42] [error] [aws_credentials] STS assume role request failed [2024/05/26 19:19:42] [ warn] [aws_credentials] No cached credentials are available and a credential refresh is already in progress. The current co-routine will retry. [2024/05/26 19:19:42] [error] [signv4] Provider returned no credentials, service=s3 [2024/05/26 19:19:42] [error] [aws_client] could not sign request [2024/05/26 19:19:42] [error] [aws_credentials] STS assume role request failed