ElasticSearch7 create unlimited amount of indices

[astelmashenko]

Describe the bug After certain amount of time shards problem raised. We checked ES and found out around 2300 shards, we names like conductor_task_log_20221004.

Details Conductor version: 3.12.2 Persistence implementation: Postgres Queue implementation: Postgres Lock: Redis

To Reproduce Steps to reproduce the behavior:

1. Just use elastic for a while

Expected behavior Data is cleaned up periodically.

[astelmashenko]

Investigating code I see

    private void createIndexesTemplates() {
        try {
            initIndexesTemplates();
            updateIndexesNames();
            Executors.newScheduledThreadPool(1).scheduleAtFixedRate(this::updateIndexesNames, 0, 1, TimeUnit.HOURS);
        } catch (Exception e) {
            logger.error("Error creating index templates!", e);
        }
    }

See this line Executors.newScheduledThreadPool(1).scheduleAtFixedRate(this::updateIndexesNames, 0, 1, TimeUnit.HOURS);

Where updateIndexesNames creates new index every week. Can someone explain why does it change names for indices?

I'd like to change this behavior, because we reaching limits on shards.

[astelmashenko]

I also checked below methods are not used anywhere:

IndexDAO.getEventExecutions
IndexDAO.getMessages

so it is probably safe to stop indexing them:

conductor.app.eventMessageIndexingEnabled=false
conductor.app.eventExecutionIndexingEnabled=false

However to stop creating new indices we need to change this config: conductor.elasticsearch.autoIndexManagementEnabled=false And provision indices for conductor manually.

Can anybody comment on this?

[manan164]

Hi @astelmashenko , Thanks for reporting this. We will add support for archival from ES also. Let me know if this works, or we can chat here for more realtime collaboration.

[astelmashenko]

@manan164 ,

We will add support for archival from ES

Yes, it will help. Additionally, I'd add reaction to indexing flag, e.g. if conductor.app.eventMessageIndexingEnabled=false is set then do not create new indices once a week. Maybe think also about changing period of creating new indices, e.g. WEEK, MONTH, YEAR Thanks

[astelmashenko]

@manan164 , are there any work done on archival of indicies?