agent crashes if kafka_exporter fails to connect to kafka

[7840vz]

What's wrong?

Agent stops immediately if agent fails to connect to kafka on start.

I think this is not right behavior. Because config is valid, and it is just networking issue, other metrics/integrations/log collector should continue working while kafka_integration should try to reconnect instead of killing the agent.

Steps to reproduce

Add config like this:

integrations:
  kafka_exporter:
    enabled: true
    kafka_uris:
      - localhost:9092
    scrape_integration: true
    scrape_interval: 15s

System information

Linux

Software version

v0.34.3

Configuration

No response

Logs

Jul 21 13:35:52 mon-1 systemd[1]: grafana-agent.service: Main process exited, code=exited, status=1/FAILURE
Jul 21 13:35:52 mon-1 systemd[1]: grafana-agent.service: Failed with result 'exit-code'.
Jul 21 13:35:52 mon-1 systemd[1]: grafana-agent.service: Scheduled restart job, restart counter is at 5.
Jul 21 13:35:52 mon-1 systemd[1]: Stopped Grafana Agent.
Jul 21 13:35:52 mon-1 systemd[1]: grafana-agent.service: Start request repeated too quickly.
Jul 21 13:35:52 mon-1 systemd[1]: grafana-agent.service: Failed with result 'exit-code'.
Jul 21 13:35:52 mon-1 systemd[1]: Failed to start Grafana Agent.
Jul 21 13:36:20 mon-1 systemd[1]: Started Grafana Agent.
Jul 21 13:36:21 mon-1 grafana-agent[34170]: ts=2023-07-21T13:36:21.717477616Z caller=exporter.go:214 level=error integration=kafka_exporter msg="Error initiating kafka client: %s" err="kafka: client has run out of available brokers to talk to: dial tcp 127.0.0.1:9092: connect: connection refused"
Jul 21 13:36:21 mon-1 grafana-agent[34170]: ts=2023-07-21T13:36:21.719833625Z caller=manager.go:261 level=error msg="failed to initialize integration. it will not run or be scraped" integration=kafka_exporter err="could not instantiate kafka lag exporter: kafka: client has run out of available brokers to talk to: dial tcp 127.0.0.1:9092: connect: connection refused"
Jul 21 13:36:21 mon-1 grafana-agent[34170]: ts=2023-07-21T13:36:21.723370642Z caller=main.go:72 level=error msg="error creating the agent server entrypoint" err="failed applying config: not all integrations were correctly updated"
Jul 21 13:36:21 mon-1 systemd[1]: grafana-agent.service: Main process exited, code=exited, status=1/FAILURE
Jul 21 13:36:21 mon-1 systemd[1]: grafana-agent.service: Failed with result 'exit-code'.