Unable to delete messages from SQS

[batflarrow]

Describe the bug

When trying to send the delete message command to sqs it randomly fails to delete for some message with the following error:

CredentialsProviderError: Could not load credentials from any providers
  File "/usr/src/app/node_modules/@aws-sdk/credential-provider-node/dist-cjs/index.js", line 25, col 19, in providers
    throw new property_provider_1.CredentialsProviderError("Could not load credentials from any providers", false);
  File "/usr/src/app/node_modules/@aws-sdk/property-provider/dist-cjs/chain.js", line 11, col 28, in <anonymous>
    return provider();
  File "/usr/src/app/node_modules/@aws-sdk/credential-provider-node/dist-cjs/index.js", in runMicrotasks
  File "node:internal/process/task_queues", line 96, col 5, in processTicksAndRejections
  File "/usr/src/app/node_modules/@aws-sdk/property-provider/dist-cjs/memoize.js", line 13, col 24, in coalesceProvider
    resolved = await pending;
  File "/usr/src/app/node_modules/@aws-sdk/property-provider/dist-cjs/memoize.js", line 32, col 24, in SignatureV4.credentialProvider
    resolved = await coalesceProvider();
  File "/usr/src/app/node_modules/@aws-sdk/signature-v4/dist-cjs/SignatureV4.js", line 84, col 29, in SignatureV4.signRequest
    const credentials = await this.credentialProvider();
  File "/usr/src/app/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js", line 13, col 18, in <anonymous>
    request: await signer.sign(args.request, {
  File "/usr/src/app/node_modules/@aws-sdk/middleware-retry/dist-cjs/StandardRetryStrategy.js", line 51, col 46, in StandardRetryStrategy.retry
    const { response, output } = await next(args);
  File "/usr/src/app/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js", line 6, col 22, in <anonymous>
    const response = await next(args);

Expected Behavior

We should be able to delete messages from the queue every time.

Current Behavior

We aren't able to delete messages from the queue and so we are forced to reprocess the messages.

Reproduction Steps

To send the delete message command we are using the following code.

await this.awsConfig.sqsClient.send(
            new DeleteMessageCommand({
              QueueUrl: process.env.QUEUE_URL,
              ReceiptHandle: message.ReceiptHandle
            })
          );

where aws config is defined in the following way

import { Injectable } from '@nestjs/common';
import { SQSClient } from '@aws-sdk/client-sqs';

@Injectable()
export class AwsConfigService {
  constructor() {
    if (process.env.NODE_ENV === 'development') process.env['NODE_TLS_REJECT_UNAUTHORIZED'] = '0';
  }

  get AWS_REGION() {
    return process.env.AWS_REGION;
  }

  get EVENTS_QUEUE_URL() {
    return process.env.QUEUE_URL;
  }

  get AWS_ENV() {
    const awsEnv: {
      region: string;
      endpoint?: string;
    } = {
      region: this.AWS_REGION
    };
    if (process.env.NODE_ENV === 'development') awsEnv.endpoint = process.env.AWS_ENDPOINT;
    return awsEnv;
  }

  get sqsClient() {
    return new SQSClient(this.AWS_ENV);
  }
}

Possible Solution

No response

Additional Information/Context

We are using the nestjs for our server and the service is being deployed to ecs. The ecs has a Iam policy to allow access to sqs. The queue being used is a FIFO queue

SDK version used

3.22.0

Environment details (OS name and version, etc.)

Ecs cluster with Launch type Fargate and platform version 1.4.0

[yenfryherrerafeliz]

Hi @batflarrow, thanks for reaching out. By looking at the error that you provided, seems like at the time the request is being made not credential provider was available, and that caused the request to fail. That could be caused by different reasons, but something you could try is to provide the credential provider manually at the moment you create the instance of the client, and you can do that by following the next steps: Note: Lets assume that the credentials are being read from a shared credentials file located at ~/.aws/credential, which credentials provider will be fromIni.

• Install the package that comes with the different credential providers:

  • yarn add @aws-sdk/credential-providers
  • or with npm
  • npm install @aws-sdk/credential-provider

• Then, when you instance your client you should do something as follow:

import {fromIni} from "@aws-sdk/credential-provider";

const client = new SQSClient({
    region: 'REGION',
    credentials: fromIni(),
}); 

Please let me know if you have any questions.

Thanks!

[batflarrow]

Hi @batflarrow, thanks for reaching out. By looking at the error that you provided, seems like at the time the request is being made not credential provider was available, and that caused the request to fail. That could be caused by different reasons, but something you could try is to provide the credential provider manually at the moment you create the instance of the client, and you can do that by following the next steps: Note: Lets assume that the credentials are being read from a shared credentials file located at ~/.aws/credential, which credentials provider will be fromIni.

• Install the package that comes with the different credential providers:

  • yarn add @aws-sdk/credential-providers
  • or with npm
  • npm install @aws-sdk/credential-provider

• Then, when you instance your client you should do something as follow:

import {fromIni} from "@aws-sdk/credential-provider";

const client = new SQSClient({
    region: 'REGION',
    credentials: fromIni(),
}); 

Please let me know if you have any questions.

Thanks!

Currently we are using IAM policy to provide the credentials so should this be added in addition to that? Also assuming our service is deployed in ecs then should we store credentials at ~/.aws/credentialin the ecs iteself?

[yenfryherrerafeliz]

Currently we are using IAM policy to provide the credentials so should this be added in addition to that? Also assuming our service is deployed in ecs then should we store credentials at ~/.aws/credentialin the ecs iteself?

Hi @batflarrow, so in that case you the credential provider should be falling in imds, so, in that case, you can do the following:

import {fromContainerMetadata} from "@aws-sdk/credential-provider";

const client = new SQSClient({
    region: 'REGION',
    credentials: fromContainerMetadata(),
}); 

Please let me know if that work for you.

Thanks!

[github-actions[bot]]

This issue has not received a response in 1 week. If you still think there is a problem, please leave a comment to avoid the issue from automatically closing.

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.