Env variable substitution not working when using target allocator and custom scrape-config

[cmergenthaler]

Component(s)

collector, target allocator

What happened?

Description

When using target allocator together with custom scrape configs, env variable substitution on the collector does not seem to work properly.

Steps to Reproduce

• Having a remote prometheus metrics endpoint, which should be scraped by custom scrape-config
• Scrape target in otel-collector is configured using env variables
• Env variables are set on otel-collector and target-allocator

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: remotepromendpoint
          scheme: https
          basic_auth:
            username: "${env::USER_ENV}"
            password_file: /etc/my-mounted-password
          tls_config:
            insecure_skip_verify: true
          static_configs:
            - targets: [ "${env::URL_ENV}" ]

Expected Result

Env variables are substituted by otel-collector and/or target-allocator

Actual Result

Otel-collector tries to scrape https://${env::URL_ENV}/metrics

Kubernetes Version

1.26.6

Operator version

0.85.0

Collector version

0.85.0

Environment information

No response

Log output

No response

Additional context

No response

[jaronoff97]

@anunarapureddy any ideas what may be happening here? I have a few theories, but don't have a ton of time to look in to this currently.

[anunarapureddy]

@jaronoff97 I can take a look into this issue.

[cole-christensen]

I am having the same issue, looks like it has been a while since there was any activity here

[jaronoff97]

@cole-christensen Can you link your configuration and explain what isn't working? Also share any logs from the operator, collector, target allocator that may be relevant when you get a chance. thank you!

[maithilee25]

Hi @anunarapureddy @jaronoff97 , Is there any update on how to resolve this?

[anunarapureddy]

Hi @anunarapureddy @jaronoff97 , Is there any update on how to resolve this?

I will take a into this issue this week and provide an update.

[atoulme]

Is there a reason for the double ::? I believe it should be https://${env:URL_ENV}/metrics

[JorTurFer]

I can confirm this case. From my super newbie side, who should replace the env vars? The target allocator before exposing the value or the collector after reading the value? I mean, can the collector get values from SD that it parses using their own env vars or the target allocator has to expose the config already parsed? (basically, where have to be the env vars placed?)

I've tried setting the env vars in both sides (without success) but I'm willing to take a look to help with this issue once I'm sure about which component should replace the env vars