mountpoint-s3
mountpoint-s3 copied to clipboard
awslabs
Assuming role configured in AWS profile does not work outside of `aws` partition (China, US Gov Cloud, etc.)
Mountpoint for Amazon S3 version
mount-s3 1.6.0
AWS Region
cn-northwest-1
Describe the running environment
When I use the command AWS_DEFAULT_REGION=cn-northwest-1 AWS_PROFILE=bwmtest2 mount-s3 --region cn-northwest-1 --debug --log-directory . / testtmppp /mnt/ mount-s3 will fail when mounted
[root@VM-0-5-centos .aws]# AWS_DEFAULT_REGION=cn-northwest-1 AWS_PROFILE=bwmtest2 mount-s3 --region cn-northwest-1 --debug --log-directory ./ testtmppp /mnt/
Error: Failed to create S3 client
Caused by:
0: initial ListObjectsV2 failed for bucket testtmppp in region cn-northwest-1
1: Client error
2: No signing credentials found
Error: Failed to create mount process
But the fact that I was able to get the IAM role correctly
AWS_DEFAULT_REGION=cn-northwest-1 AWS_PROFILE=bwmtest2 aws sts get-caller-identity
{
"Account": "714736990101",
"UserId": "AROA2M2NE66K5E6KBFALA:botocore-session-1713966126",
"Arn": "arn:aws-cn:sts::714736990101:assumed-role/s3fullrole/botocore-session-1713966126"
}
Mountpoint options
AWS_DEFAULT_REGION=cn-northwest-1 AWS_PROFILE=bwmtest2 mount-s3 --region cn-northwest-1 --debug --log-directory ./ testtmppp /mnt/
What happened?
Note that this problem only occurs when I need to ASSUME, if I use ak/sk directly I don't have this problem! This is my credentials file.
[default]
aws_access_key_id = ak
aws_secret_access_key = sk
[bwmtest2]
output=json
region=cn-northwest-1
role_arn=arn:aws-cn:iam::714736990101:role/s3fullrole
source_profile=default
I think this is a bug in China, the resources in China are segregated from global, when I refer to a source_profile mount-s3 in credentials it sends the request to global's sts endpoint instead of China's endpoint, thus causing the authentication to fail.
The steps to reproduce this are very simple in the China region, through the source_profile assume role can appear this
I saw through a packet grab that the request from sts was sent to sts.amazonaws.com instead of the China endpoint I uploaded the attachment. sts.zip
Relevant log output
2024-04-24T13:40:16.462962Z INFO mountpoint_s3::cli: mount-s3 1.6.0
2024-04-24T13:40:16.463094Z DEBUG mountpoint_s3::cli: CliArgs { bucket_name: "testtmppp", mount_point: "/mnt/", prefix: None, region: Some("cn-northwest-1"), endpoint_url: None, force_path_style: false, transfer_acceleration: false, dual_stack: false, requester_pays: false, bucket_type: None, no_sign_request: false, profile: None, read_only: false, storage_class: None, allow_delete: false, allow_overwrite: false, auto_unmount: false, allow_root: false, allow_other: false, maximum_throughput_gbps: None, max_threads: 16, part_size: 8388608, uid: None, gid: None, dir_mode: None, file_mode: None, foreground: false, expected_bucket_owner: None, log_directory: Some("./"), log_metrics: false, debug: true, debug_crt: false, no_log: false, cache: None, metadata_ttl: None, max_cache_size: None, user_agent_prefix: None, sse: None, sse_kms_key_id: None }
2024-04-24T13:40:18.465168Z ERROR awscrt::socket: id=0x7f04040023e0 fd=11: timed out, shutting down.
2024-04-24T13:40:18.465287Z ERROR awscrt::channel-bootstrap: id=0x55daf004b580: Connection failed with error_code 1048.
2024-04-24T13:40:18.465300Z ERROR awscrt::http-connection: static: Client connection failed with error 1048 (AWS_IO_SOCKET_TIMEOUT).
2024-04-24T13:40:18.465315Z WARN awscrt::connection-manager: id=0x55daeffdf900: Failed to obtain new connection from http layer, error 1048(socket operation timed out.)
2024-04-24T13:40:18.465334Z WARN awscrt::connection-manager: id=0x55daeffdf900: Failed to complete connection acquisition with error_code 1048(socket operation timed out.)
2024-04-24T13:40:18.465355Z WARN awscrt::Unknown: id=0x55daeffd4650: IMDS Client failed to acquire a connection, error code 1048(socket operation timed out.)
2024-04-24T13:40:18.465378Z ERROR awscrt::Unknown: (id=0x55daeffd4650) IMDS client failed to update the token from IMDS.
2024-04-24T13:40:20.565746Z ERROR awscrt::socket: id=0x7f0404002a30 fd=11: timed out, shutting down.
2024-04-24T13:40:20.565827Z ERROR awscrt::channel-bootstrap: id=0x55daf004b580: Connection failed with error_code 1048.
2024-04-24T13:40:20.565836Z ERROR awscrt::http-connection: static: Client connection failed with error 1048 (AWS_IO_SOCKET_TIMEOUT).
2024-04-24T13:40:20.565842Z WARN awscrt::connection-manager: id=0x55daeffdf900: Failed to obtain new connection from http layer, error 1048(socket operation timed out.)
2024-04-24T13:40:20.565852Z WARN awscrt::connection-manager: id=0x55daeffdf900: Failed to complete connection acquisition with error_code 1048(socket operation timed out.)
2024-04-24T13:40:20.565857Z WARN awscrt::Unknown: id=0x55daeffd4650: IMDS Client failed to acquire a connection, error code 1048(socket operation timed out.)
2024-04-24T13:40:20.565940Z DEBUG mountpoint_s3_client::instance_info: EC2 instance info not retrieved: ImdsQueryFailed(CrtError(Error(1048, "aws-c-io: AWS_IO_SOCKET_TIMEOUT, socket operation timed out.")))
2024-04-24T13:40:20.565989Z WARN mountpoint_s3::cli: failed to detect network throughput. Using 10 gbps as throughput. Use --maximum-throughput-gbps CLI flag to configure a target throughput appropriate for the instance. Detection failed due to: failed to get instance type: IMDS query failed: Unknown CRT error
2024-04-24T13:40:20.566013Z INFO mountpoint_s3::cli: target network throughput 10 Gbps
2024-04-24T13:40:20.566164Z TRACE mountpoint_s3_client::s3_crt_client: constructing client with auth config Default
2024-04-24T13:40:20.574065Z ERROR awscrt::AuthCredentialsProvider: Failed to resolve either region, role arn or token file path during sts web identity provider initialization.
2024-04-24T13:40:20.574112Z ERROR awscrt::AuthCredentialsProvider: Failed to resolve config profile during process credentials provider initialization.
2024-04-24T13:40:20.584323Z TRACE mountpoint_s3_client::s3_crt_client: resolved endpoint uri=Uri("https://testtmppp.s3.cn-northwest-1.amazonaws.com.cn")
2024-04-24T13:40:20.584392Z TRACE mountpoint_s3_client::s3_crt_client: resolved auth scheme auth_scheme=AuthScheme { disable_double_encoding: true, scheme_name: SigV4, signing_name: "s3", signing_region: "cn-northwest-1" }
2024-04-24T13:40:20.584645Z DEBUG list_objects{id=0 bucket="testtmppp" continued=false delimiter="" max_keys="0" prefix=""}: mountpoint_s3_client::s3_crt_client::list_objects: new request
2024-04-24T13:40:21.463395Z INFO mountpoint_s3::metrics: process.memory_usage: 10850304
2024-04-24T13:40:21.463421Z INFO mountpoint_s3::metrics: s3.client.buffer_pool.mem_limit: 2013265920
2024-04-24T13:40:21.463425Z INFO mountpoint_s3::metrics: s3.client.buffer_pool.primary_allocated: 0
2024-04-24T13:40:21.463428Z INFO mountpoint_s3::metrics: s3.client.buffer_pool.primary_cutoff: 33554432
2024-04-24T13:40:21.463431Z INFO mountpoint_s3::metrics: s3.client.buffer_pool.primary_num_blocks: 0
2024-04-24T13:40:21.463434Z INFO mountpoint_s3::metrics: s3.client.buffer_pool.primary_reserved: 0
2024-04-24T13:40:21.463436Z INFO mountpoint_s3::metrics: s3.client.buffer_pool.primary_used: 0
2024-04-24T13:40:21.463439Z INFO mountpoint_s3::metrics: s3.client.buffer_pool.secondary_reserved: 0
2024-04-24T13:40:21.463442Z INFO mountpoint_s3::metrics: s3.client.buffer_pool.secondary_used: 0
2024-04-24T13:40:21.463445Z INFO mountpoint_s3::metrics: s3.client.num_auto_default_network_io: 0
2024-04-24T13:40:21.463447Z INFO mountpoint_s3::metrics: s3.client.num_auto_ranged_copy_network_io: 0
2024-04-24T13:40:21.463450Z INFO mountpoint_s3::metrics: s3.client.num_auto_ranged_get_network_io: 0
2024-04-24T13:40:21.463453Z INFO mountpoint_s3::metrics: s3.client.num_auto_ranged_put_network_io: 0
2024-04-24T13:40:21.463456Z INFO mountpoint_s3::metrics: s3.client.num_requests_being_prepared: 0
2024-04-24T13:40:21.463458Z INFO mountpoint_s3::metrics: s3.client.num_requests_being_processed: 0
2024-04-24T13:40:21.463461Z INFO mountpoint_s3::metrics: s3.client.num_requests_stream_queued_waiting: 0
2024-04-24T13:40:21.463464Z INFO mountpoint_s3::metrics: s3.client.num_requests_streaming_response: 0
2024-04-24T13:40:21.463466Z INFO mountpoint_s3::metrics: s3.client.num_total_network_io: 0
2024-04-24T13:40:21.463469Z INFO mountpoint_s3::metrics: s3.client.request_queue_size: 0
2024-04-24T13:40:21.463472Z INFO mountpoint_s3::metrics: system.available_memory: 1708064768
2024-04-24T13:40:23.358447Z ERROR awscrt::socket: id=0x7f03fc02aad0 fd=13: timed out, shutting down.
2024-04-24T13:40:23.358524Z ERROR awscrt::channel-bootstrap: id=0x55daf004c340: Connection failed with error_code 1048.
2024-04-24T13:40:23.358533Z ERROR awscrt::http-connection: static: Client connection failed with error 1048 (AWS_IO_SOCKET_TIMEOUT).
2024-04-24T13:40:23.358539Z WARN awscrt::connection-manager: id=0x55daf011bfd0: Failed to obtain new connection from http layer, error 1048(socket operation timed out.)
2024-04-24T13:40:23.358564Z WARN awscrt::connection-manager: id=0x55daf011bfd0: Failed to complete connection acquisition with error_code 1048(socket operation timed out.)
2024-04-24T13:40:23.358575Z WARN awscrt::Unknown: id=0x55daf012a5a0: IMDS Client failed to acquire a connection, error code 1048(socket operation timed out.)
2024-04-24T13:40:23.358598Z ERROR awscrt::Unknown: (id=0x55daf012a5a0) IMDS client failed to update the token from IMDS.
2024-04-24T13:40:25.458932Z ERROR awscrt::socket: id=0x7f03fc02aad0 fd=13: timed out, shutting down.
2024-04-24T13:40:25.459014Z ERROR awscrt::channel-bootstrap: id=0x55daf004c340: Connection failed with error_code 1048.
2024-04-24T13:40:25.460350Z ERROR awscrt::http-connection: static: Client connection failed with error 1048 (AWS_IO_SOCKET_TIMEOUT).
2024-04-24T13:40:25.460361Z WARN awscrt::connection-manager: id=0x55daf011bfd0: Failed to obtain new connection from http layer, error 1048(socket operation timed out.)
2024-04-24T13:40:25.460373Z WARN awscrt::connection-manager: id=0x55daf011bfd0: Failed to complete connection acquisition with error_code 1048(socket operation timed out.)
2024-04-24T13:40:25.460377Z WARN awscrt::Unknown: id=0x55daf012a5a0: IMDS Client failed to acquire a connection, error code 1048(socket operation timed out.)
2024-04-24T13:40:25.460402Z ERROR awscrt::AuthCredentialsProvider: (id=0x55daf004c190) Default chain credentials provider failed to source credentials with error 6153(aws-c-auth: AWS_AUTH_CREDENTIALS_PROVIDER_IMDS_SOURCE_FAILURE, Valid credentials could not be sourced by the IMDS provider)
2024-04-24T13:40:25.460411Z ERROR awscrt::AuthSigning: (id=0x7f0408000d80) Credentials Provider failed to source credentials with error 6153(aws-c-auth: AWS_AUTH_CREDENTIALS_PROVIDER_IMDS_SOURCE_FAILURE, Valid credentials could not be sourced by the IMDS provider)
2024-04-24T13:40:25.460417Z ERROR awscrt::S3MetaRequest: id=0x55daf02e30f0 Meta request could not sign HTTP request due to error code 6146 (Attempt to sign an http request without credentials)
2024-04-24T13:40:25.460422Z ERROR awscrt::S3MetaRequest: id=0x55daf02e30f0 Could not prepare request 0x7f03fc000a80 due to error 6146 (Attempt to sign an http request without credentials).
2024-04-24T13:40:25.460517Z DEBUG list_objects{id=0 bucket="testtmppp" continued=false delimiter="" max_keys="0" prefix=""}: mountpoint_s3_client::s3_crt_client: S3 request failed request_type=Default crt_error=None range=None duration=4.875575188s ttfb=None request_id=<unknown>
@
To add to this, when I enable IMDS authentication I seem to be able to find the correct endpoint , but when I disable IMDS versions v1 and v2 the failure occurs!
Thanks for the bug report, @csy97. Yes, it looks like a bug. Specifically, it looks like assuming a role is not working outside of the global regions - i.e. China, US Gov Cloud.
We'll continue looking into this - I'll provide an update soon.
hello ,I noticed that this aws-c-auth seems to have fixed the problem and the project has updated the branch information for crt dependency, I would like to enquire when the new binary/rpm version of mount-s3 will be released!
Hi @csy97, we have integrated the fix and we are adding integration tests for it. We plan to include it in the next release, but we cannot share a date for it, yet.
Hey @csy97.
For this issue, we recommend that you put the configuration for the AWS profile in your AWS config file (typically ~/.aws/config) while the actual credentials live in your AWS credentials file (typically ~/.aws/credentials). (Or better yet, use short-term credentials if possible for this use case.)
For instance, we'd recommend this:
# in ~/.aws/config`
# "profile" below is important
[profile bwmtest2]
output=json
region=cn-northwest-1
role_arn=arn:aws-cn:iam::714736990101:role/s3fullrole
source_profile=default
# in ~/.aws/credentials`
[default]
aws_access_key_id = ak
aws_secret_access_key = sk
The benefit here is that you could share the config file and avoid sharing the credentials file.
I've just tested this use case in cn-north-1 with mountpoint-s3 v1.7.0 released today, storing the profile in a config file and credentials in a credentials file and it worked OK. Storing the profile configuration in the credentials file does not work, and we would not recommend it.
I'll close this issue for now, please re-open if you are still having issues.