OpenMetadata icon indicating copy to clipboard operation
OpenMetadata copied to clipboard

Published 20 hours ago • verified publisher icon open-metadata

Template for elasticsearch

Open stbogdan77 opened this issue 1 year ago • 4 comments

Hello. Is it possible to group indexes by templates when working with ElasticSearch data assets? F.e. There are indexes for every day with the same package of fields. So I would like not to describe it day by day again. I see that it is possible to get index templates in https://datahubproject.io/docs/generated/ingestion/sources/elasticsearch (ingest_index_templates: True) imageWe want to see an implementation similar to the datahub implementation. This would be very convenient. We talk about this https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html And this is a widely used case in ELK

stbogdan77 avatar Mar 13 '24 07:03 stbogdan77

It would be nice to see this feature, especially since competitors have it.

deathalt avatar Mar 13 '24 08:03 deathalt

I wonder why this feature hasn't done before. It really popular scenario to manage indices in ELK. So without this feature you must to copy metadata for every endex day without any automatisation. And of cource its impossible to use 30 indicies (i mean one index for each day) in lineage

TeXNiZ0 avatar Mar 13 '24 09:03 TeXNiZ0

@deathalt competitiors do not have all the OpenMetadata features either :) We introduced search indexes in 1.2.0 and we will evolve those features. @stbogdan77 @TeXNiZ0 it would be helpful to get some details with examples, instead of posting datahub's screenshots. If I understand the feature request correctly

  1. Give option to fetch index templates
  2. Give option to fetch index templates or individual docs . so that users can pick one or either . Do you see any reason to have both of them ingested at the same time?

harshach avatar Mar 14 '24 15:03 harshach

@harshach @TeXNiZ0 Hello. To start with, it would be great to add the ability to ingest index templates. In the future, they can be updated if the community wants to add documents and other features. Templates are needed primarily because indexes are updated regularly, making it impossible to create descriptions and lineage for them as they change every week, whereas templates remain unchanged.

stbogdan77 avatar Mar 27 '24 08:03 stbogdan77

@harshsoni2024 Lets prioritize this and work with @ulixius9 on what needs to be done

harshach avatar Aug 10 '24 17:08 harshach