Missing process instance completion and output variables in elasticsearch export

[amitonlentra]

Describe the bug

We are using zeebe 8.2.12 in self-managed mode. When enabling the elasticsearch exporter in the helm file, we see the process completion event and output variables aren't present in any of the indices created in elasticsearch. There could be more such data missing that I haven't come across

To Reproduce

1. Enable elasticsearch exporter in helm file
2. Review all the documents in the created indices

Expected behavior

Elasticsearch should have all the data about the process instance execution like other exporters do have - e.g. hazelcast-exporter.

Environment:

• OS: Linux
• Zeebe Version: 8.2.12
• Configuration: elasticsearch exporter

[koevskinikola]

Hi @amitonlentra, thank you for opening this GitHub issue.

Can you provide any additional details on how you configure the Elasticsearch exporter through the Camunda Helm chart?

One thing to note is that process events by default are not exported by the Elasticsearch exporter. You can find the default configuration values here.

If you would like to export process events, you should set the ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_INDEX_PROCESSEVENT environment variable to true. Does this fix your issue?

[amitonlentra]

@koevskinikola - thanks for looking into this. I had tried enabling all the disabled indexes but it didn't help. Please refer to the helm file as below

If you would like to export process events, you should set the ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_INDEX_PROCESSEVENT environment variable to true. Does this fix your issue?

No it didn't.

global:
  identity:
    auth:
      enabled: false     
  image:
    tag: 8.2.12

identity:
  enabled: false

optimize:
  enabled: false
  
tasklist:
  enabled: false


operate:
  enabled: false
  
elasticsearch:
  enabled: true
  replicas: 1
  image: "elasticsearch"
  imageTag: "7.17.5"
  resources:
    requests:
      cpu: 1
      memory: 4Gi
    limits:
      cpu: 1
      memory: 4Gi

retentionPolicy:
  enabled: true
  zeebeIndexTTL: 1
  operateIndexTTL: 1

connectors:
  enabled: false

zeebe:
  clusterSize: 1
  partitionCount: 1
  replicationFactor: 1
  cpuThreadCount: 4
  ioThreadCount: 4
  affinity:
    podAntiAffinity: null
  env:
    - name: ZEEBE_BROKER_EXECUTION_METRICS_EXPORTER_ENABLED
      value: "true"
    - name: ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_INDEX_PROCESSEVENT
      value: "true"
    - name: ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_INDEX_COMMAND
      value: "true"
    - name: ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_INDEX_CHECKPOINT
      value: "true"  
  pvcSize: 128Gi
  resources:
    requests:
      cpu: 1
      memory: 512Mi
    limits:
      cpu: 1
      memory: 512Mi

zeebe-gateway:
  replicas: 1
  affinity:
    podAntiAffinity: null
  env:
    - name: ZEEBE_GATEWAY_THREADS_MANAGEMENTTHREADS
      value: "4"
    - name: ZEEBE_GATEWAY_MONITORING_ENABLED
      value: "true"
  resources:
    requests:
      cpu: 1
      memory: 512Mi
    limits:
      cpu: 1
      memory: 512Mi

[amitonlentra]

@koevskinikola @romansmirnov - any inputs on this? We are blocked because of this issue. Can you please reproduce it at your end and confirm if this is not a configuration issue?

[koevskinikola]

Hi @amitonlentra, unfortunately, I haven't had time to look into this issue and try to reproduce it. I will try to dedicate some time on this next week.

In case you are a Camunda enterprise customer, it would help to prioritize this issue by opening a support ticket. If you're a Camunda community user, you can also check out the Camunda forum, in case another user encountered the same issue before.

[amitonlentra]

@koevskinikola - thanks for the update.

If you're a Camunda community user, you can also check out the Camunda forum, in case another user encountered the same issue before.

I am community user and I had checked on the forums before raising this issue. No one has faced this (probably no one checked the data in elasticsearch till now) which surprises me and as per my understanding Operate and Optimize rely on this exporter to know about workflow executions. The documentation also doesn't mention anything about incompleteness. All these facts worry me and make me doubt if I am missing anything in the configuration.

The impact of this not working is huge for us because we need to now look out for other exporters which increases the effort and time to test new components.

[amitonlentra]

@koevskinikola - is there a test class in the zeebe source code that you can point me to that can simulate this use case? Basically a test that executes a bpmn process with elasticsearch exporter enabled. I can debug this flow and see what the issue is.

I tried looking into a few test classes but wasn't able to find an end-to-end test.

[amitonlentra]

@koevskinikola - we troubleshooted this issue by running the standalone broker locally and connecting to a locally running elasticsearch server. Surprising while debugging we see the end event, intermediate event and script task records coming to the export method of ElasticsearchExporter but they aren't physically present in the index when we query through elasticsearch REST api. There are no exceptions too. The code execution flow is the same for all types of records. We event tried upgrading elasticsearch to 8.12.2 but that made no difference.

Any clues on how can we troubleshoot this issue further?

[amitonlentra]

Sorry this turned out to be a false alarm. Ultimately it was an issue with the REST API query string. By default it returns only 10 documents. Since we weren't able to find the documents we were looking for, we thought the issue is with the indexing logic. Adding the size parameter in the query showed all the documents