Getting a `failed to load IMDS session token` on a non-AWS environment

[fulmicoton]

observed on scaleway with the following config

version: 0.7

metastore_uri: s3://qw-data-xxxxxx
default_index_root_uri: s3://qw-data-xxxxxx

storage:
  s3:
    access_key_id: xxxxxxx
    secret_access_key: yyyyyyyy
    region: fr-par
    endpoint: https://s3.fr-par.scw.cloud
    force_path_style_access: ${QW_S3_FORCE_PATH_STYLE_ACCESS:-false}
    disable_multi_object_delete: false
    disable_multipart_upload: false

indexer:
  enable_otlp_endpoint: ${QW_ENABLE_OTLP_ENDPOINT:-true}

jaeger:
  enable_endpoint: ${QW_ENABLE_JAEGER_ENDPOINT:-true}

As reported by @ineumann

^[[2m2024-09-16T09:47:44.250Z^[[0m ^[[33m WARN^[[0m ^[[2maws_config::imds::region^[[0m^[[2m:^[[0m failed to load region from IMDS ^[[3merr^[[0m^[[2m=^[[0mfailed to load IMDS session token: dispatch failure: timeout: error trying to connect: HTTP connect timeout occurred after 1s: HTTP connect timeout occurred after 1s: timed out (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: hyper::Error(Connect, HttpTimeoutError { kind: "HTTP connect", duration: 1s }), connection: Unknown } }) }))
^[[2m2024-09-16T09:47:45.252Z^[[0m ^[[33m WARN^[[0m ^[[2maws_config::imds::region^[[0m^[[2m:^[[0m failed to load region from IMDS ^[[3merr^[[0m^[[2m=^[[0mfailed to load IMDS session token: dispatch failure: timeout: error trying to connect: HTTP connect timeout occurred after 1s: HTTP connect timeout occurred after 1s: timed out (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: hyper::Error(Connect, HttpTimeoutError { kind: "HTTP connect", duration: 1s }), connection: Unknown } }) }))
^[[2m2024-09-16T09:49:22.920Z^[[0m ^[[31mERROR^[[0m ^[[2mquickwit_proto::error^[[0m^[[2m:^[[0m gRPC transport error: Timeout expired ^[[3mcode^[[0m^[[2m=^[[0mCancelled ^[[3mrpc^[[0m^[[2m=^[[0m"publish_splits"
^[[2m2024-09-16T09:49:22.921Z^[[0m ^[[31mERROR^[[0m ^[[2mquickwit_actors::spawn_builder^[[0m^[[2m:^[[0m actor-failure ^[[3mcause^[[0m^[[2m=^[[0mfailed to publish splits

Could be similar to https://github.com/delta-io/delta-rs/pull/2817/files

[idrissneumann]

Hi.

This dashboard might help to have more details (last occurrences of the error, some stats): https://grafana.comwork.io/public-dashboards/801a34eb04a5462d968e5f2d3a5b3e49

Thanks

[miton18]

Same warning using CleverCloud Cellar

2025-03-05T15:20:37.774Z  WARN imds_load_region: aws_config::imds::region: failed to load region from IMDS err=failed to load IMDS session token: dispatch failure: timeout: error trying to connect: HTTP connect timeout occurred after 1s: HTTP connect timeout occurred after 1s: timed out (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: hyper::Error(Connect, HttpTimeoutError { kind: "HTTP connect", duration: 1s }), connection: Unknown } }) }))

[miton18]

Setting AWS_EC2_METADATA_DISABLED=true seems to fix

[ysfmkr]

Setting AWS_EC2_METADATA_DISABLED=true seems to fix

@miton18 where this configuration come from? I've searched the code and documentation and didn't find this config I try to run on kubernetes cluster in Alicloud, is this config still required?