Importing data from Elasticsearch

[saig0]

• import the data that is provided by the Elasticsearch exporter

[asaf]

@saig0 first, i'd like to say that it's a great effort as there's currently no way to read Zeebe data easily.

I would like to clarify this, when ES exporter is enabled, Operate uses ES transformers in order to create its own data, so data gets built on the fly when it is pushed by the exporter.

Since the approach you chose is to create a local DB, I was wondering what would be the approach to support importing data from ES.

[saig0]

@asaf, my idea is similar to Operate.

• pull ES
• aggregate data
• store data in internal DB (which is used to query from)
• pull next data from ES

The difference to Operate is that the data is stored in a database that is maybe not ES.

[asaf]

@saig0 ah, I thought operate puts some "processors" on ES somehow, yea fair enough, I guess you just need some kind of cursor / store the last document fetched as with Hazelcast sequence

[mohit2494]

Hi there, I'm looking forward to contribute to this through Hacktoberfest. Can someone help me out chart the details, so that I can make incremental code reviews for this. Best :)

[saig0]

@mohit2494 awesome :tada:

The basic idea here is to import the data from ES and store it similar to the Hazelcast importer.

• create a new module for the ES importer similar to /hazelcast-importer
• create a new importer to poll the data from the different ES indexes and store them in the same way as /hazelcast-importer/src/main/kotlin/io/zeebe/zeeqs/importer/hazelcast/HazelcastImporter.kt
• store the current position of the ES index to continue after a restart

The data in ES are generated by the ES exporter: https://github.com/zeebe-io/zeebe/tree/develop/exporters/elasticsearch-exporter Here are the templates for the ES indexes: https://github.com/zeebe-io/zeebe/tree/develop/exporters/elasticsearch-exporter/src/main/resources

Please ask if you've any question or need help :)

[mohit2494]

Thank you so much for the reply. It looks interesting, though a bit challenging. Can I have an email id where I can get in touch.

Best. Mohit

[saig0]

@mohit2494 you can get in touch using the Zeebe Slack channel. You can ask questions in general or contact me directly.

[fromz]

@saig0 is there any idea of an ETA for this at all?

[saig0]

@fromz no ETA. Currently, I don't have any plans to work on it because I don't have a concrete need for it.

But I'm open to contributions from the community. :rocket: