The "remote.s3" component throws error since grafana/agent:v0.39.0

[rparkhomchuk]

What's wrong?

Staring from grafana/agent:v0.39.0 the "remote.s3" component started to throw "not found, ResolveEndpointV2" error. It works perfectly fine v0.38.*.

The issue could be related to the new version of aws-sdk-go.

Steps to reproduce

To reproduce the issue start Grafana Agent in Flow mode with below configuration:

root_config.river

remote.s3 "metrics" {
  path = "s3://<bucket>>/metrics.river"
  poll_frequency = "1m"
  is_secret = false
}

module.string "metrics" {
  content = remote.s3.metrics.content

  arguments {
    username = env("PROMETHEUS_USERNAME")
    password = env("PROMETHEUS_PASSWORD")
  }
}

prometheus.exporter.unix "default" { }

prometheus.scrape "local_agent" {
  targets         = prometheus.exporter.unix.default.targets
  forward_to      = [module.string.metrics.exports.prometheus_remote_write.receiver]
  scrape_interval = "10s"
}

s3 config

argument "username" { }

argument "password" { }

export "prometheus_remote_write" {
  value = prometheus.remote_write.grafana_cloud
}

prometheus.remote_write "grafana_cloud" {
    endpoint {
        url = "https://prometheus-prod-01-eu-west-0.grafana.net/api/prom/push"

        basic_auth {
          username = argument.username.value
          password = argument.password.value
        }
    }
}

System information

No response

Software version

No response

Configuration

No response

Logs

No response

[hainenber]

Hi @rparkhomchuk, can you help building a Flow binary from the linked PR to test out the fix? make agent-flow and you're good to go :D

Apologies for the bizarre request as I don't have AWS account personally.

[rparkhomchuk]

Hi @rparkhomchuk, can you help building a Flow binary from the linked PR to test out the fix? make agent-flow and you're good to go :D

Apologies for the bizarre request as I don't have AWS account personally.

Hi @hainenber, I was able to build a binary, but getting 404 if execute the ./build/grafana-agent-flow run --server.http.listen-addr=0.0.0.0:12345 ./build/config.river command.

UI page loads successfully if I run the agent in Flow mode from the image built with make agent-image. The issue still exists, I am still getting the not found, ResolveEndpointV2 error during remote.s3 component loading.

[hainenber]

Thanks for your revert! This is a bit bothering but can you help checking again if the aws-sdk-go-v2/* modules are up to latest? I checked the linked issue, the proposed solution and other linked PRs and ppl managed to resolve it once everything aws-sdk-go-v2-related got upgraded to latest 🤔

You can extract the Agent's binary in the built image and run go version -m /path/to/agent/binary to check the modules's version.

Here's mine

If it still shows the error, I think I'd go with downgrading instead to avoid knocking on your doors :D

[rparkhomchuk]

Hi @hainenber, looks like I've built an image without your changes :) All works for me from image containing your changes. Thanks

[tpaschalis]

Thank you for the confirmation @rparkhomchuk (as well as the original catch) 🙌