ECK Filebeat default user cannot create indicess (403 - unauthorised error)

[akumar-99]

Bug Report

What did you do? I am running ECK on AWS EKS 1.24. My use case is straight I want to scrape the container logs using filebeat and send them to their respective indices based on the container name.

What did you expect to see? Indices should get created and I should see entries. It is able to get the data but not able to create index and send it.

What did you see instead? Under which circumstances? I see and error in the filebeat pods.

{"log.level":"warn","@timestamp":"2023-07-11T07:47:09.824Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":446},"message":"Cannot index event publisher.Event{Content:beat.Event{--data--}, Flags:0x1, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=403): {\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/bulk[s]] is unauthorized for user [observability-filebeat-beat-user] with effective roles [beats_admin,eck_beat_es_filebeat_role_v77,ingest_admin,kibana_admin,remote_monitoring_agent] on indices [production-pod-xyz-8.8.2], this action is granted by the index privileges [create_doc,create,delete,index,write,all]\"}, dropping event!","service.name":"filebeat","ecs.version":"1.6.0"}

Environment

• ECK version: 2.8.0

• Kubernetes information: AWS EKS 1.24

  insert any information about your Kubernetes environment that could help us:

  • On premise ?
  • Cloud: GKE / EKS / AKS ?
  • Kubernetes distribution: Openshift / Rancher / PKS ?

  for each of them please give us the version you are using

$ kubectl version

clientVersion:
  buildDate: "2023-06-14T09:53:42Z"
  compiler: gc
  gitCommit: 25b4e43193bcda6c7328a6d147b1fb73a33f1598
  gitTreeState: clean
  gitVersion: v1.27.3
  goVersion: go1.20.5
  major: "1"
  minor: "27"
  platform: linux/amd64
kustomizeVersion: v5.0.1
serverVersion:
  buildDate: "2023-05-22T23:41:27Z"
  compiler: gc
  gitCommit: 05d192f0de17608d98e17761ad3cffa9a6407f2f
  gitTreeState: clean
  gitVersion: v1.24.14-eks-c12679a
  goVersion: go1.19.9
  major: "1"
  minor: 24+
  platform: linux/amd64

WARNING: version difference between client (1.27) and server (1.24) exceeds the supported minor version skew of +/-1

• Resource definition:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
  namespace: observability
spec:
  version: 8.8.2
  nodeSets:
  - name: master
    count: 1
    config:
      node:
        roles:
          - master
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
        storageClassName: gp3
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
          resources:
            requests:
              cpu: 100m
              memory: 2Gi
            limits:
              cpu: 500m
              memory: 2.5Gi
  - name: data
    count: 1
    config:
      node:
        roles:
          - data
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
        storageClassName: gp3
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
          resources:
            requests:
              cpu: 100m
              memory: 2Gi
            limits:
              cpu: 500m
              memory: 2Gi
---
apiVersion: enterprisesearch.k8s.elastic.co/v1
kind: EnterpriseSearch
metadata:
  name: enterprise-search
spec:
  version: 8.8.2
  count: 1
  elasticsearchRef:
    name: elasticsearch
    namespace: observability
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
  namespace: observability
spec:
  version: 8.8.2
  count: 1
  elasticsearchRef:
    name: elasticsearch
    namespace: observability
  enterpriseSearchRef:
    name: enterprise-search
    namespace: observability
---
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: filebeat
  namespace: observability
spec:
  type: filebeat
  version: 8.8.2
  elasticsearchRef:
    name: elasticsearch
    namespace: observability
  config:
    filebeat:
      autodiscover:
        providers:
          - type: kubernetes
            node: ${NODE_NAME}
            namespace: default
            templates:
              - config:
                  - type: container
                    paths:
                      - "/var/log/containers/*-${data.kubernetes.container.id}.log"
    output:
      elasticsearch:
        index: "production-pod-%{[kubernetes.container.name]}-%{[agent.version]}"
    
    setup.template:
      enabled: true
      name: 'production-pod'
      pattern: 'production-pod-*'
      index: "production-pod-*"
      ilm.enabled: false

  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: elastic-beat-filebeat
        automountServiceAccountToken: true
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true
        containers:
        - name: filebeat
          securityContext:
            runAsUser: 0
            privileged: true
          resources:
            limits:
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 100Mi
          env:
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          volumeMounts:
          - name: varlogcontainers
            mountPath: /var/log/containers
          - name: varlogpods
            mountPath: /var/log/pods
        volumes:
        - name: varlogcontainers
          hostPath:
            path: /var/log/containers
        - name: varlogpods
          hostPath:
            path: /var/log/pods
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: elastic-beat-autodiscover
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - namespaces
  - pods
  - nodes
  verbs:
  - get
  - watch
  - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: elastic-beat-filebeat
  namespace: observability
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: elastic-beat-autodiscover-binding
subjects:
- kind: ServiceAccount
  name: elastic-beat-filebeat
  namespace: observability
roleRef:
  kind: ClusterRole
  name: elastic-beat-autodiscover
  apiGroup: rbac.authorization.k8s.io

• Logs:

{"log.level":"warn","@timestamp":"2023-07-11T07:47:09.824Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":446},"message":"Cannot index event publisher.Event{Content:beat.Event{--data--}, Flags:0x1, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=403): {\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/bulk[s]] is unauthorized for user [observability-filebeat-beat-user] with effective roles [beats_admin,eck_beat_es_filebeat_role_v77,ingest_admin,kibana_admin,remote_monitoring_agent] on indices [production-pod-xyz-8.8.2], this action is granted by the index privileges [create_doc,create,delete,index,write,all]\"}, dropping event!","service.name":"filebeat","ecs.version":"1.6.0"}

[intermittentnrg]

Also asked here without answer: https://discuss.elastic.co/t/filebeat-running-on-eck-cannot-create-an-index/316464

Seems a new role is needed as ECK doesn't allow editing the default/system roles https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-users-and-roles.html#k8s_creating_custom_roles

So just create user / roles either through Kibana or link above. Then ECK filebeat - remove elasticsearchRef and add the below:

spec:
  config:
    filebeat:
      output:
        elasticsearch:
          hosts:
            - http://elasticsearch-es-http.elastic.svc:9200
          password: xxxxx
          username: xxxx

custom outputs documented here: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-beat-configuration.html#k8s-beat-set-beat-output

Hope this helps someone I didn't find this simple well documented.