Can't authenticate with SQS queue when using pod service account!

[Daniel-Canaf]

Hi, we intend on using the plugin at scale, on multiple EKS pods. However, assigning the pod with a service account which is associated with an IAM role fails to authenticate with the SQS queue. We've followed the procedure as described here: https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html The Logstash.conf is as follows:

input: |- s3snssqs { region => "us-east-1" s3_default_options => { "endpoint_discovery" => true } queue => "https://sqs.us-east-1.amazonaws.com/<ACCOUNT_ID>/sqs-logstash-test" queue_owner_aws_account_id => "<ACCOUNT_ID>" type => "elb" from_sns => false sqs_skip_delete => false ecs_compatibility => disabled }

I've also verified the pod's ENV vars:

AWS_DEFAULT_REGION: us-east-1 AWS_REGION: us-east-1 AWS_ROLE_ARN: arn:aws:iam::<ACCOUNT_ID>:role/logstashs-on-kubernetes AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token

The role has the required trusted entities, and the policies (Allow sqs + s3) work as expected with an EC2 instances who assumed a similar role.

The error we get when using this pod is :

Registering SQS input {:queue=>"https://sqs.us-east-1.amazonaws.com/<....>/sqs-logstash-test"} [2021-12-05T09:12:29,903][INFO ][logstash.inputs.s3snssqs ][main] [[main]-pipeline-manager] connected to queue. {:queue_url=>"https://sqs.us-east-1.amazonaws.com/<..>/sqs-logstash-test"} [2021-12-05T09:12:29,943][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"} .... ..... [2021-12-05T09:12:31,136][WARN ][logstash.inputs.s3snssqs ][main][1636db8eb3fb87ac06ea0b26cf44e8bb468a57708952d726ee505fe791060286] Aws::SQS::Errors::ServiceError ... retrying SQS request with exponential backoff {:queue=>"https://sqs.us-east-1.amazonaws.com/<...>/sqs-logstash-test", :sleep_time=>1, :error=>#<Aws::SQS::Errors::AccessDenied: Access to the resource https://sqs.us-east-1.amazonaws.com/<...>/sqs-logstash-test is denied.>}

IAM access advisor shows the policy for SQS hasn't been accessed.

Is this functionality supposed to work? If not, what are our options for giving the pod the required permissions? Thanks

[christianherweg0807]

The problem is the old aws_sdk used in logstash. I pushed elastic to update logstash_mixin_aws...

[kalavathiy]

[2023-03-13T19:36:34,355][WARN ][logstash.inputs.s3snssqs ][main][a9a137fb82bd2c5f96099381cd651662f321fc38016f399a9b47f41ece68be48] Aws::SQS::Errors::ServiceError ... retrying SQS request with exponential backoff {:queue=>"https://sqs.us-west-2.amazonaws.com//sqs-lsp-test", :sleep_time=>1, :error=>#<Aws::SQS::Errors::AccessDenied: Access to the resource https://sqs.us-west-2.amazonaws.com//sqs-lsp-test is denied.>}