geesefs icon indicating copy to clipboard operation
geesefs copied to clipboard
verified publisher icon yandex-cloud

AWS IAM authorization with non-standard profile

Open Felixoid opened this issue 1 year ago • 6 comments

Is there a way to automatically retrieve the security credentials, when the SG is not default?

$ ./geesefs-linux-amd64 --iam --iam-flavor=imdsv1 bucket:prefix directory
s3.INFO Bad response while trying to get IAM token from http://169.254.169.254/latest/meta-data/iam/security-credentials/default: invalid character '<' looking for beginning of value
main.FATAL Mounting file system: Unable to access 'github-runners-data': bucket github-runners-data does not exist
$ curl http://169.254.169.254/latest/meta-data/iam/security-credentials/
another_profile
$ ./geesefs-linux-amd64 --iam --iam-flavor=imdsv1 --iam-url=http://169.254.169.254/latest/meta-data/iam/security-credentials/another_profile --endpoint=https://s3.amazonaws.com clickhouse-builds debug
s3.INFO Successfully acquired IAM Token
s3.INFO Falling back to v2 signer
main.FATAL Mounting file system: Unable to access 'clickhouse-builds': Forbidden: Forbidden
        status code: 403, request id: 021W1F45Q183ZFYJ, host id: 56fmmo...

It still fails for some reason, but it's another issue, I think

Felixoid avatar Dec 05 '24 13:12 Felixoid

Hi I think you also need to set --iam-header, the default is for yandex s3 :)

vitalif avatar Dec 05 '24 22:12 vitalif

I tried adding --iam-header=X-Amz-Security-Token, but doesn't help so far

$ curl -LO https://github.com/yandex-cloud/geesefs/releases/download/v0.42.3/geesefs-linux-amd64
$ chmod +x geesefs-linux-amd64
$ ./geesefs-linux-amd64 --iam --iam-flavor=imdsv1 --iam-url=http://169.254.169.254/latest/meta-data/iam/security-credentials/$PROFILE --iam-header=X-Amz-Security-Token --endpoint=https://s3.amazonaws.com --debug_s3 clickhouse-builds debug |& sed 's/IQoJb3Jp.*/IQoJb3Jp......../'
s3.INFO Successfully acquired IAM Token
s3.DEBUG HEAD https://s3.amazonaws.com/clickhouse-builds = 403 [us-east-1]
s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /clickhouse-builds/xyn165r43zzkbxopc2jg8ivez20z6evf HTTP/1.1
Host: s3.amazonaws.com
User-Agent: GeeseFS/0.42.3 (go1.22.9; linux; amd64)
X-Amz-Security-Token: IQoJb3Jp........


-----------------------------------------------------
s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Tue, 10 Dec 2024 16:59:23 GMT
Server: AmazonS3
X-Amz-Id-2: dQk1ThFrbqBJadHLBXEy0UEuR/lHqKCwywelO7EYGQTHybgL1Wcok3HdN1IoXxs3QFtYzaaj+vs=
X-Amz-Request-Id: W0BJ8KBVYJS0F8KM



-----------------------------------------------------
s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, attempt 0/3, error Forbidden: Forbidden
        status code: 403, request id: W0BJ8KBVYJS0F8KM, host id: dQk1ThFrbqBJadHLBXEy0UEuR/lHqKCwywelO7EYGQTHybgL1Wcok3HdN1IoXxs3QFtYzaaj+vs=
s3.INFO Falling back to v2 signer
s3.DEBUG DEBUG: Request s3/HeadObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /clickhouse-builds/xyn165r43zzkbxopc2jg8ivez20z6evf HTTP/1.1
Host: s3.amazonaws.com
User-Agent: GeeseFS/0.42.3 (go1.22.9; linux; amd64)
X-Amz-Security-Token: IQoJb3Jp........


-----------------------------------------------------
s3.DEBUG DEBUG: Response s3/HeadObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Tue, 10 Dec 2024 16:59:23 GMT
Server: AmazonS3
X-Amz-Id-2: 8w8PDrovl08CTdKE9zaPtaE+p4ql6iHslMMyf4kH+mbfrDyOeKRWtukCUYCEAx8oX2+k//2tkB8=
X-Amz-Request-Id: W0BXXJ4TGHHBCEVJ



-----------------------------------------------------
s3.DEBUG DEBUG: Validate Response s3/HeadObject failed, attempt 0/3, error Forbidden: Forbidden
        status code: 403, request id: W0BXXJ4TGHHBCEVJ, host id: 8w8PDrovl08CTdKE9zaPtaE+p4ql6iHslMMyf4kH+mbfrDyOeKRWtukCUYCEAx8oX2+k//2tkB8=
main.FATAL Mounting file system: Unable to access 'clickhouse-builds': Forbidden: Forbidden
        status code: 403, request id: W0BXXJ4TGHHBCEVJ, host id: 8w8PDrovl08CTdKE9zaPtaE+p4ql6iHslMMyf4kH+mbfrDyOeKRWtukCUYCEAx8oX2+k//2tkB8=

Felixoid avatar Dec 10 '24 17:12 Felixoid

Ok, I'm not sure if amazon actually supports authentication using just the access token like yandex. Maybe they require both access&secret key + the token? Because their documentation says that you "must include the token in signature" here https://docs.aws.amazon.com/AmazonS3/latest/API/RESTAuthentication.html#UsingTemporarySecurityCredentials , and the signature is generated using static keys. Can you check the content of http://169.254.169.254/latest/meta-data/iam/security-credentials/$PROFILE manually? In Yandex case it has keys Code, Token, and Expiration, and that's all. Maybe amazon also returns access&secret key there?

vitalif avatar Dec 13 '24 10:12 vitalif

Sure, here it is:

{
  "Code": "Success",
  "LastUpdated": "2024-12-13T10:29:23Z",
  "Type": "AWS-HMAC",
  "AccessKeyId": "ASI...",
  "SecretAccessKey": "XU...",
  "Token": "IQoJb3J..",
  "Expiration": "2024-12-13T17:04:35Z"
}

And here's the part of aws s3 ls --debug request to see, how it is implemented in the aws-cli

$ aws s3 ls --debug s3://BUCKET
2024-12-13 10:40:03,906 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.21.3 Python/3.12.6 Linux/6.5.0-1020-aws exe/x86_64.ubuntu.22
2024-12-13 10:40:03,906 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', 'ls', '--debug', 's3://BUCKET']
2024-12-13 10:40:03,917 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_s3 at 0x7fb3e78be020>
2024-12-13 10:40:03,917 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_ddb at 0x7fb3e7df5620>
2024-12-13 10:40:03,917 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>>
2024-12-13 10:40:03,917 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7fb3e7d6b6a0>
2024-12-13 10:40:03,917 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7fb3e7d88b80>
2024-12-13 10:40:03,917 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x7fb3e78d4900>
2024-12-13 10:40:03,917 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_history_commands at 0x7fb3e793c2c0>
2024-12-13 10:40:03,917 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>>
2024-12-13 10:40:03,917 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_waiters at 0x7fb3e78d47c0>
2024-12-13 10:40:03,917 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x7fb3e7785df0>>
2024-12-13 10:40:03,917 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.21.3/dist/awscli/data/cli.json
2024-12-13 10:40:03,919 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_types at 0x7fb3e78018a0>
2024-12-13 10:40:03,919 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function no_sign_request at 0x7fb3e7801bc0>
2024-12-13 10:40:03,919 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_verify_ssl at 0x7fb3e7801b20>
2024-12-13 10:40:03,919 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_read_timeout at 0x7fb3e7801d00>
2024-12-13 10:40:03,920 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_connect_timeout at 0x7fb3e7801c60>
2024-12-13 10:40:03,920 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <built-in method update of dict object at 0x7fb3e779b180>
2024-12-13 10:40:03,920 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.21.3 Python/3.12.6 Linux/6.5.0-1020-aws exe/x86_64.ubuntu.22
2024-12-13 10:40:03,920 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', 'ls', '--debug', 's3://BUCKET']
2024-12-13 10:40:03,920 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_timestamp_parser at 0x7fb3e78be8e0>
2024-12-13 10:40:03,920 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x7fb3e8efa480>
2024-12-13 10:40:03,921 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_binary_formatter at 0x7fb3e77689a0>
2024-12-13 10:40:03,921 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function no_pager_handler at 0x7fb3e8a4e700>
2024-12-13 10:40:03,921 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7fb3e87a1440>
2024-12-13 10:40:03,922 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2024-12-13 10:40:03,924 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x7fb3e792c9a0>
2024-12-13 10:40:03,924 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_json_file_cache at 0x7fb3e7de1a80>
2024-12-13 10:40:03,924 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <function add_waiters at 0x7fb3e78d47c0>
2024-12-13 10:40:03,924 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x7fb3e7785df0>>
2024-12-13 10:40:03,925 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3_ls: calling handler <function add_waiters at 0x7fb3e78d47c0>
2024-12-13 10:40:03,925 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3_ls: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x7fb3e7785df0>>
2024-12-13 10:40:03,925 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.paths: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fb3e88b6e10>
2024-12-13 10:40:03,925 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fb3e88b6e10>
2024-12-13 10:40:03,925 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.page-size: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fb3e88b6e10>
2024-12-13 10:40:03,925 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.human-readable: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fb3e88b6e10>
2024-12-13 10:40:03,925 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.ls: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fb3e8a5bda0>
2024-12-13 10:40:03,926 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.summarize: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fb3e88b6e10>
2024-12-13 10:40:03,926 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.ls: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fb3e8a5bda0>
2024-12-13 10:40:03,926 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.request-payer: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fb3e88b6e10>
2024-12-13 10:40:03,926 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2024-12-13 10:40:03,927 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): 169.254.169.254:80
2024-12-13 10:40:03,928 - MainThread - urllib3.connectionpool - DEBUG - http://169.254.169.254:80 "PUT /latest/api/token HTTP/1.1" 200 56
2024-12-13 10:40:03,929 - MainThread - urllib3.connectionpool - DEBUG - Resetting dropped connection: 169.254.169.254
2024-12-13 10:40:03,929 - MainThread - urllib3.connectionpool - DEBUG - http://169.254.169.254:80 "GET /latest/meta-data/placement/availability-zone/ HTTP/1.1" 200 10
2024-12-13 10:40:03,930 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2024-12-13 10:40:03,930 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2024-12-13 10:40:03,930 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity
2024-12-13 10:40:03,930 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: sso
2024-12-13 10:40:03,930 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2024-12-13 10:40:03,930 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: custom-process
2024-12-13 10:40:03,930 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: config-file
2024-12-13 10:40:03,930 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: ec2-credentials-file
2024-12-13 10:40:03,930 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: boto-config
2024-12-13 10:40:03,930 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: container-role
2024-12-13 10:40:03,930 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: iam-role
2024-12-13 10:40:03,931 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): 169.254.169.254:80
2024-12-13 10:40:03,931 - MainThread - urllib3.connectionpool - DEBUG - http://169.254.169.254:80 "PUT /latest/api/token HTTP/1.1" 200 56
2024-12-13 10:40:03,932 - MainThread - urllib3.connectionpool - DEBUG - Resetting dropped connection: 169.254.169.254
2024-12-13 10:40:03,933 - MainThread - urllib3.connectionpool - DEBUG - http://169.254.169.254:80 "GET /latest/meta-data/iam/security-credentials/ HTTP/1.1" 200 9
2024-12-13 10:40:03,933 - MainThread - urllib3.connectionpool - DEBUG - Resetting dropped connection: 169.254.169.254
2024-12-13 10:40:03,934 - MainThread - urllib3.connectionpool - DEBUG - http://169.254.169.254:80 "GET /latest/meta-data/iam/security-credentials/IAM_PROFILE HTTP/1.1" 200 1582
2024-12-13 10:40:03,936 - MainThread - botocore.credentials - DEBUG - Found credentials from IAM Role: IAM_PROFILE
2024-12-13 10:40:03,937 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.21.3/dist/awscli/botocore/data/endpoints.json
2024-12-13 10:40:03,963 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7fb3e9756c00>
2024-12-13 10:40:03,974 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.21.3/dist/awscli/botocore/data/s3/2006-03-01/service-2.json
2024-12-13 10:40:03,984 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.21.3/dist/awscli/botocore/data/s3/2006-03-01/service-2.sdk-extras.json
2024-12-13 10:40:03,994 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.21.3/dist/awscli/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json
2024-12-13 10:40:03,998 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.21.3/dist/awscli/botocore/data/partitions.json
2024-12-13 10:40:04,000 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7fb3e9e7c5e0>
2024-12-13 10:40:04,001 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7fb3e9e7c360>
2024-12-13 10:40:04,001 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for s3 via: environment_service
2024-12-13 10:40:04,001 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for s3 via: environment_global
2024-12-13 10:40:04,001 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for s3 via: config_service
2024-12-13 10:40:04,001 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for s3 via: config_global
2024-12-13 10:40:04,001 - MainThread - botocore.configprovider - DEBUG - No configured endpoint found.
2024-12-13 10:40:04,003 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2024-12-13 10:40:04,006 - MainThread - botocore.utils - DEBUG - Registering S3 region redirector handler
2024-12-13 10:40:04,006 - MainThread - botocore.utils - DEBUG - Registering S3Express Identity Resolver
2024-12-13 10:40:04,015 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.21.3/dist/awscli/botocore/data/s3/2006-03-01/paginators-1.json
2024-12-13 10:40:04,015 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.21.3/dist/awscli/botocore/data/s3/2006-03-01/paginators-1.sdk-extras.json
2024-12-13 10:40:04,015 - MainThread - botocore.hooks - DEBUG - Event provide-client-params.s3.ListObjectsV2: calling handler <function base64_decode_input_blobs at 0x7fb3e7769300>
2024-12-13 10:40:04,016 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <function set_list_objects_encoding_type_url at 0x7fb3e97867a0>
2024-12-13 10:40:04,016 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <function validate_bucket_name at 0x7fb3e9785300>
2024-12-13 10:40:04,016 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <function remove_bucket_from_url_paths_from_model at 0x7fb3e9787420>
2024-12-13 10:40:04,016 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x7fb3e658cb90>>
2024-12-13 10:40:04,016 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <bound method S3ExpressIdentityResolver.inject_signing_cache_key of <botocore.utils.S3ExpressIdentityResolver object at 0x7fb3e658ce30>>
2024-12-13 10:40:04,016 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <function generate_idempotent_uuid at 0x7fb3e9785120>
2024-12-13 10:40:04,016 - MainThread - botocore.hooks - DEBUG - Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x7fb3e9787600>
2024-12-13 10:40:04,016 - MainThread - botocore.hooks - DEBUG - Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x7fb3e658cb90>>
2024-12-13 10:40:04,016 - MainThread - botocore.regions - DEBUG - Calling endpoint provider with parameters: {'Bucket': 'BUCKET', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'ForcePathStyle': False, 'Accelerate': False, 'UseGlobalEndpoint': False, 'Prefix': '', 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
2024-12-13 10:40:04,017 - MainThread - botocore.regions - DEBUG - Endpoint provider result: https://BUCKET.s3.us-east-1.amazonaws.com
2024-12-13 10:40:04,017 - MainThread - botocore.regions - DEBUG - Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None"
2024-12-13 10:40:04,017 - MainThread - botocore.regions - DEBUG - Selected auth type "v4" as "v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}
2024-12-13 10:40:04,018 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListObjectsV2: calling handler <function add_expect_header at 0x7fb3e97856c0>
2024-12-13 10:40:04,018 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListObjectsV2: calling handler <bound method S3ExpressIdentityResolver.apply_signing_cache_key of <botocore.utils.S3ExpressIdentityResolver object at 0x7fb3e658ce30>>
2024-12-13 10:40:04,018 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListObjectsV2: calling handler <function inject_api_version_header_if_needed at 0x7fb3e9786c00>
2024-12-13 10:40:04,018 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=ListObjectsV2) with params: {'url_path': '?list-type=2', 'query_string': {'prefix': '', 'delimiter': '/', 'encoding-type': 'url'}, 'method': 'GET', 'headers': {'User-Agent': 'aws-cli/2.21.3 md/awscrt#0.22.0 ua/2.0 os/linux#6.5.0-1020-aws md/arch#x86_64 lang/python#3.12.6 md/pyimpl#CPython cfg/retry-mode#standard md/installer#exe md/distrib#ubuntu.22 md/prompt#off md/command#s3.ls'}, 'body': b'', 'auth_path': '/BUCKET?list-type=2', 'url': 'https://BUCKET.s3.us-east-1.amazonaws.com/?list-type=2&prefix=&delimiter=%2F&encoding-type=url', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7fb3e652a8a0>, 'has_streaming_input': False, 'auth_type': 'v4', 'unsigned_payload': None, 'encoding_type_auto_set': True, 's3_redirect': {'redirected': False, 'bucket': 'BUCKET', 'params': {'Bucket': 'BUCKET', 'Prefix': '', 'Delimiter': '/', 'EncodingType': 'url'}}, 'S3Express': {'bucket_name': 'BUCKET'}, 'signing': {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}, 'endpoint_properties': {'authSchemes': [{'disableDoubleEncoding': True, 'name': 'sigv4', 'signingName': 's3', 'signingRegion': 'us-east-1'}]}}}
2024-12-13 10:40:04,018 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.ListObjectsV2: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fb3e66e24b0>>
2024-12-13 10:40:04,018 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.ListObjectsV2: calling handler <function set_operation_specific_signer at 0x7fb3e9784f40>
2024-12-13 10:40:04,018 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.ListObjectsV2: calling handler <function remove_arn_from_signing_path at 0x7fb3e9787560>
2024-12-13 10:40:04,018 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.ListObjectsV2: calling handler <bound method S3ExpressIdentityResolver.resolve_s3express_identity of <botocore.utils.S3ExpressIdentityResolver object at 0x7fb3e658ce30>>
2024-12-13 10:40:04,019 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2024-12-13 10:40:04,019 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
GET
/
delimiter=%2F&encoding-type=url&list-type=2&prefix=
host:BUCKET.s3.us-east-1.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20241213T104004Z
x-amz-security-token:IQoJb3J...

host;x-amz-content-sha256;x-amz-date;x-amz-security-token
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2024-12-13 10:40:04,019 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20241213T104004Z
20241213/us-east-1/s3/aws4_request
ef5b26911675244995caf11ddb7cb26dba9d9eddd03b1eb01007c9a7da1f00a8
2024-12-13 10:40:04,019 - MainThread - botocore.auth - DEBUG - Signature:
7c6e059e135f231f2b30a58fd8703d49dbd2ecd5509f48baca48b8ee3513a7b3
2024-12-13 10:40:04,019 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://BUCKET.s3.us-east-1.amazonaws.com/?list-type=2&prefix=&delimiter=%2F&encoding-type=url, headers={'User-Agent': b'aws-cli/2.21.3 md/awscrt#0.22.0 ua/2.0 os/linux#6.5.0-1020-aws md/arch#x86_64 lang/python#3.12.6 md/pyimpl#CPython cfg/retry-mode#standard md/installer#exe md/distrib#ubuntu.22 md/prompt#off md/command#s3.ls', 'X-Amz-Date': b'20241213T104004Z', 'X-Amz-Security-Token': b'IQoJb3J...', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=ASIA{ACCESS_KEY_IR}/20241213/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=7c6e059e135f231f2b30a58fd8703d49dbd2ecd5509f48baca48b8ee3513a7b3'}>
2024-12-13 10:40:04,019 - MainThread - botocore.httpsession - DEBUG - Certificate path: /usr/local/aws-cli/v2/2.21.3/dist/awscli/botocore/cacert.pem
2024-12-13 10:40:04,020 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): BUCKET.s3.us-east-1.amazonaws.com:443
2024-12-13 10:40:04,070 - MainThread - urllib3.connectionpool - DEBUG - https://BUCKET.s3.us-east-1.amazonaws.com:443 "GET /?list-type=2&prefix=&delimiter=%2F&encoding-type=url HTTP/1.1" 200 None
2024-12-13 10:40:04,071 - MainThread - botocore.hooks - DEBUG - Event before-parse.s3.ListObjectsV2: calling handler <function _handle_200_error at 0x7fb3e9787880>
2024-12-13 10:40:04,071 - MainThread - botocore.hooks - DEBUG - Event before-parse.s3.ListObjectsV2: calling handler <function handle_expires_header at 0x7fb3e97876a0>
2024-12-13 10:40:04,071 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': '9+82PVFoOKbC7DqB6bOXF0BG7vfD9hj+4qX5p+KeoqDXWYseigMhTIDt/aAmGCm7gcwvI3c7Hhc=', 'x-amz-request-id': 'DQ3X93MVZCVWGXWK', 'Date': 'Fri, 13 Dec 2024 10:40:05 GMT', 'x-amz-bucket-region': 'us-east-1', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Server': 'AmazonS3'}
2024-12-13 10:40:04,071 - MainThread - botocore.parsers - DEBUG - Response body:
b'<?xml version="1.0" encoding="UTF-8"?>\n<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>BUCKET</Name><Prefix></Prefix><KeyCount>27</KeyCount>........'
2024-12-13 10:40:04,072 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <function _update_status_code at 0x7fb3e97879c0>
2024-12-13 10:40:04,072 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7fb3e658ce00>>
2024-12-13 10:40:04,072 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2024-12-13 10:40:04,072 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7fb3e658cb90>>
2024-12-13 10:40:04,072 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjectsV2: calling handler <function decode_list_object_v2 at 0x7fb3e97868e0>
2024-12-13 10:40:04,073 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjectsV2: calling handler <function enhance_error_msg at 0x7fb3e78be480>
2024-12-13 10:40:04,073 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjectsV2: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7fb3e658cb60>>
                           PRE 23.3/ ....................

'Authorization' header in the request very much looks like a Credential=ASIA{ACCESS_KEY_IR}

Felixoid avatar Dec 13 '24 10:12 Felixoid

Ok, just as I suspected, they want both token & access keys. Ok, I can add this implementation too :-)

vitalif avatar Dec 13 '24 12:12 vitalif

With automatic IAM profile resolution? :star_struck:

Felixoid avatar Dec 13 '24 12:12 Felixoid