[Filebeat] Journald input doesn't work in container

[SlavikCA]

I'm running filebeat on the Harvester Kubernetes cluster:

helm repo add elastic https://helm.elastic.co
helm repo update
helm install filebeat elastic/filebeat -n elk-stack -f logs/beats-values.yaml

with following values:

daemonset:
  enabled: true

  extraVolumes:
    - name: journald-logs
      hostPath:
        path: /run/log/journal/
        type: Directory
  extraVolumeMounts:
    - name: journald-logs
      mountPath: /run/log/journal/

  hostNetworking: true

  # Allows you to add any config files in /usr/share/filebeat such as filebeat.yml for daemonset
  filebeatConfig:
    filebeat.yml: |
      filebeat.inputs:
      
      - type: journald
        seek: head
        paths:
          - /var/log/journal/
          - /run/log/journal/

      output.elasticsearch:
        hosts: 'es01:9200'
        username: '${ELASTICSEARCH_USERNAME}'
        password: '${ELASTICSEARCH_PASSWORD}'
        protocol: http

  secretMounts: []

  securityContext:
    # - User that the container will execute as. Typically necessary to run as root (0) in order to properly collect host container logs.
    runAsUser: 0
    # - Whether to execute the Filebeat containers as privileged containers. Typically not necessarily unless running within environments such as OpenShift.
    privileged: true

deployment:
  enabled: false

Nothing sent. What am I missing?

[SlavikCA]

I confirmed, that journal files are present inside the container:

kubectl exec -it filebeat-filebeat-w6wkx  -n elk-stack -- ls -lhR /run/log/journal
...
-rw-r----- 1 root 483 8.0M Oct  2 02:46 system.journal

 kubectl exec -it filebeat-filebeat-w6wkx  -n elk-stack -- ls -lhR /var/log/journal
...
/var/log/journal/f93c751c5aa9e98da8a607da66d82d3c:
-rw-r----- 1 root 483 32M Oct 17 02:58 system.journal
-rw-r----- 1 root 483 32M Oct  2 02:46 [email protected]~

Is it expected, that I should be able to run journalctl inside the container? Because I can't:

kubectl exec -it filebeat-filebeat-w6wkx  -n elk-stack --  journalctl 
error: Internal error occurred: error executing command in container: 
failed to exec in container: failed to start exec "b215f1d5ca6d80bc52f2d1b0221ad3a262b803dd15709e7f181c5b00386085ed": 
OCI runtime exec failed: exec failed: unable to start container process: 
exec: "journalctl": executable file not found in $PATH: unknown

[nihr43]

I second this; I'm running 'Beat' crd 8.16.0 under eck-operator 2.14.0, getting image docker.elastic.co/beats/filebeat:8.16.0; trying to use the journald input type to get kernel logs. I too am seeing this in the daemonset logs:

Input 'journald' failed with: input kernel failed: could not start journal reader: cannot start journalctl: exec: \"journalctl\": executable file not found in $PATH

[Lebvanih]

We are facing the same issue. Is there any plan to address this issue and allow filebeat in container to use the journald logs input again?

[lpeter91]

Also having basically the same issue just with the Elastic Agent container. There's no systemd (and thus journalctl) in that image either.

[passing]

as a workaround, you can create and use your own docker image like this:

FROM bin.private.zooplus.net/beats/filebeat:8.17.1
USER root
RUN apt-get install -y systemd
USER 1000

but preferably this gets included in the official image

[hamelg]

You need to provide an image with a journalctl compatible with the host running the container. With openshift we build a custom image from the toolbox image. Here is a snippet of the buildconfig.

    dockerfile: |
      FROM registry.redhat.io/ubi9/toolbox
      USER root
      RUN \
      rpm -i https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.16.3-x86_64.rpm

Next, you need to configure the Filebeat container volume mounts according to this guide.

[georgivalentinov]

Just to increment the issues counter - this breaks the common case where Filebeat is used as a K8s DaemonSet to gather journald logs.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[belimawr]

Indeed the journalctl binary is missing in Filebeat's Docker image.

The Elastic-Agent complete image, does contain it:

docker run --rm -it --entrypoint /bin/bash  docker.elastic.co/beats/elastic-agent-complete:8.17.3
elastic-agent@56cb73617994:~$ journalctl
No journal files were found.
-- No entries --
elastic-agent@56cb73617994:~$ exit

Workaround

As @passing suggested, the workaround is to install systemd in the docker image Dockerfile

FROM docker.elastic.co/beats/filebeat:8.17.3
USER root
RUN apt-get install -y systemd
USER 1000

Build:

docker build . -t "filebeat:8.17.3"

Test:

docker run --rm -it --entrypoint /bin/bash filebeat:8.17.3                         
filebeat@e10ecc5cea60:~$ journalctl                                                      
No journal files were found.
-- No entries --
filebeat@e10ecc5cea60:~$ 

[lpeter91]

Since the system integration is probably the most commonly used one, I would argue that the elastic-agent image should also have Systemd installed. Currently the need for using the complete image for one of the more trivial integrations is not (well?) documented and I would consider it suboptimal UX.

(Especially since the complete image is almost twice as large as the basic one [3.18GB vs 1.68GB]. 3GB+ is very far from the lightweight log collectors Beats promises to be. I know I'm slightly mixing up Beats with the Agent and that Agent bulkiness is a separate issue, hence the parentheses.)

[belimawr]

Here is the PR adding journalctl to Filebeat docker image: https://github.com/elastic/beats/pull/44056

[xanadu-dev]

The release notes for filebeat 8.18.1 state "Journald input now works on Docker containers"

However when testing the latest docker image I get the error

"message":"Input 'journald' failed with: input system failed: could not start journal reader: cannot start journalctl: exec: "journalctl": executable file not found in $PATH"

[belimawr]

It later got removed from the wolfi images (see https://github.com/elastic/beats/pull/44118), anyways, I'm checking what happened/in which images it should be available.

[belimawr]

@xanadu-dev, the ubi variant contains journalctl: docker.elastic.co/beats/filebeat-ubi:8.18.1:

$ docker run --rm -it --entrypoint /bin/sh docker.elastic.co/beats/filebeat-ubi:8.18.1
sh-5.1$ journalctl
No journal files were found.
-- No entries --
sh-5.1$ exit

I'm still investigating why docker.elastic.co/beats/filebeat:8.18.1 does not contain it.

[belimawr]

I found the issue, in main (and 9.x), all docker images variant, aside from wolfi, are based on redhat/ubi9-minimal, however 8.18, probably the whole 8.x series, are based on ubuntu:24.04, so when the PR was backported the default variant docker.elastic.co/beats/filebeat:8.18.1 did not contain journalctl because we check the base image name to know which package manager to use, thus excluding the Ubuntu based ones.

I'll re-open the issue until we get the default variant in 8.18 fixed and also fix the docs/change log.

[passing]

apparently, the issue has been solved in version 8.18.3 (in addition to 9.0.1):

$ docker run --rm -it --user root --entrypoint journalctl docker.elastic.co/beats/filebeat:8.18.3 --version
systemd 255 (255.4-1ubuntu8.8)

[xanadu-dev]

It appears that this issue has recurred again with filebeat 8.19.3

Same error as previously - cannot start journalctl: exec: "journalctl": executable file not found in $PATH"

[belimawr]

It appears that this issue has recurred again with filebeat 8.19.3

Same error as previously - cannot start journalctl: exec: "journalctl": executable file not found in $PATH"

I'll take a look at it today.

[belimawr]

@xanadu-dev, indeed this is a regression :/. It looks like the backport PR to 8.19 was not created. I've manually created it and added an entry in the known issues for 8.19.x.

Here is the PR: https://github.com/elastic/beats/pull/46625