Expose IMDS Client Command

[commiterate]

Describe the feature

Expose a command for IMDS requests (e.g. aws imds get --path /latest/user-data) which automatically handles IMDS session token fetching + caching.

Use Case

Shell scripts are often used to set up EC2 instances manually (e.g. via SSH or SSM sessions) or automatically (e.g. EC2 user data scripts, CodeDeploy Agent hooks, SSM documents). They are also used for simple on-instance EC2 Auto Scaling lifecycle hook daemons (e.g. systemd service units).

These may need to fetch data from IMDS (e.g. user data, auto scaling target lifecycle state).

Today, this requires using curl to manually fetch IMDS session tokens for use in subsequent get requests.

Proposed Solution

The AWS SDKs implement an IMDS client. This is used to support IMDS region + credentials providers.

Some AWS SDKs expose the IMDS client to let users make IMDS calls without having to worry about fetching + caching session tokens. For example:

• AWS SDK for Go v2
• AWS SDK for Java 2.x
• AWS SDK for Ruby v3
• AWS SDK for Rust 0.x (via aws_config 1.x)

The idea is to do the same for the AWS SDK for Python (boto) and the AWS CLI.

1. Update botocore to make the IMDSFetcher or a similar class expose a general-purpose public get() method.
2. Add a CLI for the general-purpose public get() method.

Tangent: Rather than manually write an IMDS client in all AWS SDKs, is there a Smithy (or its internal predecessor) model describing IMDS which can be fed into the Smithy code generators?

This would probably need new Smithy auth and protocol traits like aws.auth#imdsv2 and aws.protocols#imds.

(cc: @mtdowling)

Other Information

No response

Acknowledgements

• [ ] I may be able to implement this feature request
• [ ] This feature might incur a breaking change

CLI version used

2.*

Environment details (OS name and version, etc.)

All

[imSanko]

To implement a feature that exposes a command for IMDS (Instance Metadata Service) requests in the AWS CLI and SDK for we can prefer this:

1. Update botocore to Expose IMDSFetcher

The botocore library, which is the foundation of boto3, already contains an IMDSFetcher class that handles IMDSv2 session token fetching and caching. The first step is to expose a public get() method in this class to allow general-purpose IMDS requests.

Changes in botocore:

• Add a Public get() Method: Modify the IMDSFetcher class to include a get() method that takes a path (e.g., /latest/user-data) and returns the corresponding metadata.
• Handle Token Fetching and Caching: Ensure the get() method automatically handles the fetching and caching of IMDSv2 session tokens, so users don’t need to manage this manually.
• Error Handling: Implement robust error handling for cases where the IMDS is unreachable or the requested path is invalid.

class IMDSFetcher:
    def __init__(self, timeout=None, num_attempts=None):
        # Existing initialization code
        pass

    def get(self, path):
        """
        Fetches metadata from IMDS at the specified path.
        Automatically handles IMDSv2 session token fetching and caching.
        """
        token = self._fetch_token()  # Fetches and caches the token if needed
        headers = {"X-aws-ec2-metadata-token": token}
        response = self._get_request(path, headers)
        return response

2. Add IMDS Command to AWS CLI

Next, add a new command to the AWS CLI that leverages the updated botocore functionality to make IMDS requests.

Changes in AWS CLI:

• Add imds Command: Introduce a new imds command in the AWS CLI with a get subcommand.
• Path Argument: Allow users to specify the IMDS path (e.g., /latest/user-data) as an argument.
• Output Handling: Output the fetched metadata directly to the terminal.

aws imds get --path /latest/user-data

Implementation:

• CLI Command Registration: Register the imds command in the AWS CLI command structure.
• Integration with botocore: Use the IMDSFetcher.get() method to fetch the metadata and display it.

import botocore.session

def imds_get(args):
    session = botocore.session.get_session()
    fetcher = session.create_client('imds').meta.imds_fetcher
    response = fetcher.get(args.path)
    print(response)

def main():
    parser = argparse.ArgumentParser(description='Fetch metadata from IMDS.')
    parser.add_argument('--path', required=True, help='The IMDS path to fetch.')
    args = parser.parse_args()
    imds_get(args)

if __name__ == '__main__':
    main()

3. Smithy Model for IMDS (Optional)

To standardize IMDS client implementations across AWS SDKs, consider creating a Smithy model for IMDS. This would allow SDKs to generate IMDS clients automatically, reducing the need for manual implementation.

Smithy Model:

• Define IMDS Protocol: Create a new Smithy protocol (e.g., aws.protocols#imds2) that describes the IMDS API.
• Generate SDK Code: Use the Smithy code generator to produce IMDS client code for various SDKs.

namespace aws.protocols

@protocolDefinition
structure IMDS2 {
    version: "2023-01-01"
    operations: [GetMetadata]
}

operation GetMetadata {
    input: GetMetadataInput
    output: GetMetadataOutput
}

structure GetMetadataInput {
    @required
    path: string
}

structure GetMetadataOutput {
    metadata: string
}

4. Backward Compatibility

• Deprecation Warning: If the changes introduce breaking changes, provide clear deprecation warnings and migration guides.
• Fallback Mechanism: Ensure the new get() method can handle IMDSv1 for backward compatibility if needed.

[commiterate]

Looks like an LLM-generated comment that's just taken the issue and fluffed it up with some trivial tasks and code snippets. The Smithy model is very incorrect and IMDSv1 support should not be considered as the AWS SDK IMDS clients should only support IMDSv2 as per the docs (IMDSv1 is deprecated).

[khushail]

Hi @commiterate , thanks for requesting this and having patience. I would reach out to team whether this is something being actively considered by them or get their insights on its inclusion. Will share updates.

Thanks.

[khushail]

@commiterate ,We don’t support IMDS as a public client, there isn’t anything to expose. If there is anything needed, one would have to write customization for this just like some SDKs (Java, Ruby) did. However I see , your usecase is covered through curl - https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html#instance-metadata-ex-2a So I am keeping this request open for future tracking purpose.

Thanks

[commiterate]

@khushail The Use Case section in the issue already mentions curl.

Today, this requires using curl to manually fetch IMDS session tokens for use in subsequent get requests.

Using curl isn't convenient because users need to juggle IMDS session tokens by themselves. In particular, customers need to:

1. Cache + refresh the IMDS token (recommended in IMDS docs to avoid throttling).
2. Include it as a header in every curl call.
3. Handle backoff + retries if throttled.

For example, a minimal EC2 Auto Scaling lifecycle hook daemon script that handles the above looks like this:

#!/usr/bin/env bash

# --------------------------------------------------
#
# Required IMDS boilerplate start.
#
# --------------------------------------------------

# IMDS IPv4 link-local or IPv6 unique local address.
imds_endpoint=169.254.169.254
# IMDS tokens should be valid for 60 seconds.
imds_token_ttl_seconds=60
# IMDS tokens should be refreshed if ≤ 5 seconds are left.
imds_token_ttl_buffer_seconds=5

# Current IMDS token. Assume the current time is past the Unix epoch.
imds_token=""
imds_token_expiration_timestamp=0

# Refresh IMDS token if needed.
imds_token_refresh() {
  current_timestamp=$(date +%s)
  if [[ "$imds_token_expiration_timestamp" -le $((current_timestamp + imds_token_ttl_buffer_seconds)) ]]; then
    imds_token=$(curl \
      --silent \
      --retry 2 \
      --request PUT \
      --header "X-aws-ec2-metadata-token-ttl-seconds: ${imds_token_ttl_seconds}" \
      "http://${imds_endpoint}/latest/api/token")
    imds_token_expiration_timestamp=$((current_timestamp + imds_token_ttl_seconds))
  fi
}

# Get with IMDS token refresh.
imds_get() {
  while getopts ":p:" opt; do
    case "$opt" in
      # IMDS path (e.g. `/latest/user-data`).
      p)
        path="$OPTARG"
        ;;
    esac
  done

  imds_token_refresh

  echo $(curl \
    --silent \
    --retry 2 \
    --request GET \
    --header "X-aws-ec2-metadata-token: ${imds_token}" \
    "http://${imds_endpoint}${path}")
}

# --------------------------------------------------
#
# Required IMDS boilerplate end.
#
# --------------------------------------------------

# --------------------------------------------------
#
# Optional IMDS boilerplate start.
#
# --------------------------------------------------

# Get the region.
imds_get_region() {
  echo $(imds_get -p /latest/meta-data/placement/region)
}

# Get the user data.
imds_get_user_data() {
  echo $(imds_get -p /latest/user-data)
}

# Get the auto scaling group from the user data (e.g. pretend user data is JSON).
imds_get_user_data_auto_scaling_group() {
  echo $(imds_get_user_data | jq -cr ".auto_scaling_group")
}

# Get the instance ID.
imds_get_instance_id() {
  echo $(imds_get -p /latest/meta-data/instance-id)
}

# Get the auto scaling target lifecycle state.
imds_get_auto_scaling_target_lifecycle_state() {
  echo $(imds_get -p /latest/meta-data/autoscaling/target-lifecycle-state)
}

# --------------------------------------------------
#
# Optional IMDS boilerplate end.
#
# --------------------------------------------------

graceful_shutdown_complete() {
  echo "Completing graceful shutdown."
  aws autoscaling complete-lifecycle-action \
    --region $(imds_get_region) \
    --auto-scaling-group-name $(imds_get_user_data_auto_scaling_group) \
    --instance-id $(imds_get_instance_id) \
    --lifecycle-hook-name graceful-shutdown \
    --lifecycle-action-result CONTINUE
}

# Do an instance-initiated shutdown so we're only subject
# to OS-level shutdown timeouts and not the EC2-initiated shutdown
# timeout (forces a hard shutdown after a few minutes).
#
# Auto scaling lifecycle hook timeouts can go up to 2 hours.
graceful_shutdown() {
  trap graceful_shutdown_complete SIGTERM

  echo "Starting graceful shutdown."
  /usr/bin/systemctl poweroff

  # Wait for SIGTERM.
  while true; do
    sleep 1
  done
}

# Main loop.
while true; do
  target_lifecycle_state=$(imds_get_auto_scaling_target_lifecycle_state)
  case "$(target_lifecycle_state)" in
    Detached | InService | Standby)
      ;;
    Terminated)
      graceful_shutdown
      ;;
    *)
      echo "Unhandled target lifecycle state: ${target_lifecycle_state}"
      ;;
  esac
  # 10 second polling interval.
  sleep 10
done

This boilerplate needs to be repeated across customer scripts and also doesn't share the IMDS token cache across processes unlike the AWS CLI which caches credentials in ~/.aws/cli/cache. Trying to re-implement a shared cache which the AWS CLI already does can be error prone.

Instead, it feels like this undifferentiated heavy lifting should be done by the AWS CLI.

[khushail]

@commiterate thanks for explaining that. However I am sorry to say team won't be able to work on this now.