How to Implement `fromInstanceMetadata` to Retrieve EC2 Role Credentials using the v3 SDK and IMDSv2?

[mdaronco]

Describe the issue

Hello! I'm running an app within a Node v16 Docker container (Docker version 20.10.7, build f0df350). The EC2 instance is running Ubuntu 20.04.2 LTS.

I am trying to authenticate my Node.js application with AWS. I've attached an EC2 Role to the instance that contains the necessary access policies for the services used by the app. However, I'm getting the following error when attempting to make the first AWS service call with secretsManager.getSecretValue:

2022-09-09T02:05:48:  ProviderError: Error response received from instance metadata service
    at ClientRequest.<anonymous> (/usr/src/app/node_modules/@aws-sdk/credential-provider-imds/dist-cjs/remoteProvider/httpRequest.js:26:38)
    at ClientRequest.emit (node:events:394:28)
    at HTTPParser.parserOnIncomingClient (node:_http_client:621:27)
    at HTTPParser.parserOnHeadersComplete (node:_http_common:127:17)
    at Socket.socketOnData (node:_http_client:487:22)
    at Socket.emit (node:events:394:28)
    at addChunk (node:internal/streams/readable:312:12)
    at readableAddChunk (node:internal/streams/readable:287:9)
    at Socket.Readable.push (node:internal/streams/readable:226:10)
    at TCP.onStreamRead (node:internal/stream_base_commons:190:23) {
  tryNextLink: true,
  statusCode: 401,
  '$metadata': { attempts: 1, totalRetryDelay: 0 }
}

These are the modular dependencies I've installed:

"@aws-sdk/client-s3": "^3.165.0",
"@aws-sdk/client-secrets-manager": "^3.165.0",
"@aws-sdk/credential-providers": "^3.165.0",
"@aws-sdk/s3-request-presigner": "^3.165.0",

Here is the failing code that executes during application startup:

import { fromInstanceMetadata } from '@aws-sdk/credential-providers';
import {
	S3Client,
	GetObjectCommand, GetObjectCommandInput,
	PutObjectCommandInput, PutObjectCommand,
	DeleteObjectsCommand, DeleteObjectsCommandInput
} from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
import { SecretsManager } from '@aws-sdk/client-secrets-manager';

// other imports...

export class AWSClient {
	private s3Client: S3Client;
	private secretsManager: SecretsManager;
	private static instance: AWSClient;

	static async getInstance(): Promise<AWSClient> {
		if (!AWSClient.instance) {
			AWSClient.instance = new AWSClient();
		}
		return AWSClient.instance;
	}

	constructor() {
		const region = process.env.AWS_DEFAULT_REGION || 'us-east-1';
		const credentials = fromInstanceMetadata({ timeout: 10000, maxRetries: 2, });
		
		this.s3Client = new S3Client({ apiVersion: 'latest', credentials, region, });
		this.secretsManager = new SecretsManager({ credentials, region, });
	}
	
	private getSecretKeys(SecretId: SecretId) {
		return new Promise<string>((resolve, reject) => {
			this.secretsManager.getSecretValue({ SecretId, }, (err, data) => {
				if (err) {
                                         // CODE FAILS AND PRINTS HERE
                                         console.error(err);
                                         
					if (err.code === 'DecryptionFailureException')
						// Secrets Manager can't decrypt the protected secret text using the provided KMS key.
						reject(err);
					else if (err.code === 'InternalServiceErrorException')
						// An error occurred on the server side.
						reject(err);
					else if (err.code === 'InvalidParameterException')
						// You provided an invalid value for a parameter.
						reject(err);
					else if (err.code === 'InvalidRequestException')
						// You provided a parameter value that is not valid for the current state of the resource.
						reject(err);
					else if (err.code === 'ResourceNotFoundException')
						// We can't find the resource that you asked for.
						reject(err);
				} else {
					// Decrypts secret using the associated KMS key.
					// Depending on whether the secret is a string or binary, one of these fields will be populated.
					if ('SecretString' in data!) {
						resolve(data.SecretString as string);
					} else {
						const buff = Buffer.from(data!.SecretBinary as any, 'base64');
						const decodedBinarySecret = buff.toString('ascii');
						resolve(decodedBinarySecret);
					}
				}
			});
		});
	}
	
	// other helper methods ...	
}

The loading credentials docs state you can use EC2 roles to automatically provide credentials to the SDK. Is the configuration code in the constructor above correct given an EC2 Role is attached to the EC2 instance this code is running on?

Also, I'm having no problem receiving instance metadata using IMDSv1 (when the instance's MetadataOptions.HttpTokens is set to optional and I use v2 of the SDK). However, since migrating to the Node.js v3 SDK and IMDSv2 (by setting MetadataOptions.HttpTokens to required) I've started getting the error above. I've followed the guide for configuring the instance metadata to force IMDSv2.

Any guidance would be greatly appreciated. Thank you!

Links

fromInstanceMetadata

[yenfryherrerafeliz]

Hi @mdaronco, thanks for opening this issue. Basically, by default JS V3 tries to retrieve the credentials from the instance metadata service by using IMDSv2, however there are some conditions in the logic that may cause the credentials to be retrieved by using IMDSv1. For example, If the following condition is met, then the credentials will be retrieved by using IMDSv1. Another thing to consider is, that if you are running containers on top of an EC2 instance, then you need to make sure the hop limit is set to at least 2, otherwise the credentials resolution will timeout in the first attempt and the IMDSv1 will be used to retrieve the credentials in the second+ attempts. You can find more information about this here and here.

If you have any question please just let me know.

Thanks!

[github-actions[bot]]

This issue has not received a response in 1 week. If you still think there is a problem, please leave a comment to avoid the issue from automatically closing.

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.