Alloy running in a container with loki.source.journal unable to scrape journald logs from host if path is not specified

[suever]

What's wrong?

According to the documentation for the loki.source.journal component, if no path is specified for the loki.source.journal component, then /var/log/journal is used by default .

When using the very basic configuration from the example:

loki.source.journal "LABEL" {
  forward_to    = RECEIVER_LIST
}

When running alloy with the loki.source.journal component configured as above on a host directly, logs are discovered and scraped properly; however, when running within a container on the host and mounting /var/log/journal as a volume into the container, the logs are not discovered.

After digging into the code, this is due to the fact that, when no path is specified, alloy takes a different code path which uses go-systemd's defaults for discovering the journal entries which uses the SD_JOURNAL_LOCAL_ONLY option which ensures that only logs generated on the local machine are discovered. If you explicitly specify a path configuration option to the loki.source.journal component, then journal entries are discovered a different way which does not specify the SD_JOURNAL_LOCAL_ONLY option, which does not limit it to local logs only and it is able to discover the logs from the host /var/log/journal.

I'm not sure if this is a documentation issue for the component where it should state that you must set path = "/var/log/journal" when running in Docker, or probably more appropriately a code change where the default path is specified explicitly in alloy codebase and then go-systemd/sdjournal is invoked the same way regardless of whether a path was supplied or not which removes the behavior of alloy being dependent upon go-systemd's implementation.

I am happy to contribute a fix, but wanted to let the maintainers determine what is most appropriate to avoid confusion for users like this one

Steps to reproduce

Create the following config in config.alloy

loki.source.journal "read" {
  forward_to    = [ loki.write.loki.receiver ]
  labels        = {component = "loki.source.journal"}
}

loki.write "loki" {
  endpoint {
    url = "http://loki-gateway:8080/loki/api/v1/push"
  }
}

Run it locally on a linux machine

./alloy-linux-amd64  run --server.http.listen-addr=0.0.0.0:12345 --storage.path=./alloy ./config.alloy

Ensure journal targets are discovered:

curl -s http://localhost:12345/metrics | grep 'loki_source_journal_target_lines_total{'

loki_source_journal_target_lines_total{component_id="loki.source.journal.read",component_path="/"} 2580

Run it in docker:

docker run -v /var/log/journal:/var/log/journal -v `pwd`/config.alloy:/etc/alloy/config.alloy -p 12345:12345 grafana/alloy:v1.1.0 run --server.http.listen-addr=0.0.0.0:12345 /etc/alloy/config.alloy

Ensure no journal targets are discovered

curl -s http://localhost:12345/metrics | grep 'loki_source_journal_target_lines_total{'

loki_source_journal_target_lines_total{component_id="loki.source.journal.read",component_path="/"} 0

Modify the config to explicitly set the path

loki.source.journal "read" {
  forward_to    = [ loki.write.loki.receiver ]
  labels        = {component = "loki.source.journal"}
  path          = "/var/log/journal"
}

loki.write "loki" {
  endpoint {
    url = "http://loki-gateway:8080/loki/api/v1/push"
  }
}

Re-run in docker, and ensure that targets are now discovered

curl -s http://localhost:12345/metrics | grep 'loki_source_journal_target_lines_total{'

loki_source_journal_target_lines_total{component_id="loki.source.journal.read",component_path="/"} 2580

System information

Linux / Docker

Software version

v1.1.0 and master

Configuration

loki.source.journal "read" {
  forward_to    = [ loki.write.loki.receiver ]
  labels        = {component = "loki.source.journal"}
  path = "/var/log/journal"
}

loki.write "loki" {
  endpoint {
    url = "http://loki-gateway:8080/loki/api/v1/push"
  }
}

Logs

No response