OpenMetadata
OpenMetadata copied to clipboard
open-metadata
OpenMetadata ignores elasticsearch index prefix
Affected module It affects the backend
Describe the bug
We are using a corporate managed elasticsearch cluster where I am allowed to do all operations on aliases / indices with the following schema:
alias_<project-name>-*-app-* (alias)
<project-name>-*-app-* (indices)
So I set the clusterAlias in the Helm Chart as follows:
clusterAlias: <project-name>-app-openmetadata
OpenMetadata creates the following indices:
alias index
<project-name>-app-openmetadata_AggregatedCostAnalysisReportData <project-name>-app-openmetadata_aggregated_cost_analysis_report_data_index
<project-name>-app-openmetadata_RawCostAnalysisReportData <project-name>-app-openmetadata_raw_cost_analysis_report_data_index
<project-name>-app-openmetadata_all <project-name>-app-openmetadata_classification_search_index
<project-name>-app-openmetadata_all <project-name>-app-openmetadata_container_search_index
<project-name>-app-openmetadata_all <project-name>-app-openmetadata_dashboard_data_model_search_index
<project-name>-app-openmetadata_all <project-name>-app-openmetadata_dashboard_search_index
<project-name>-app-openmetadata_all <project-name>-app-openmetadata_dashboard_service_search_index
<project-name>-app-openmetadata_all <project-name>-app-openmetadata_data_product_search_index
<project-name>-app-openmetadata_all <project-name>-app-openmetadata_database_schema_search_index
<project-name>-app-openmetadata_all <project-name>-app-openmetadata_database_search_index
<project-name>-app-openmetadata_all <project-name>-app-openmetadata_database_service_search_index
<project-name>-app-openmetadata_all <project-name>-app-openmetadata_glossary_search_index
<project-name>-app-openmetadata_all <project-name>-app-openmetadata_glossary_term_search_index
<project-name>-app-openmetadata_all <project-name>-app-openmetadata_ingestion_pipeline_search_index
<project-name>-app-openmetadata_all <project-name>-app-openmetadata_messaging_service_search_index
<project-name>-app-openmetadata_all <project-name>-app-openmetadata_mlmodel_search_index
<project-name>-app-openmetadata_all <project-name>-app-openmetadata_mlmodel_service_search_index
<project-name>-app-openmetadata_all <project-name>-app-openmetadata_pipeline_search_index
<project-name>-app-openmetadata_all <project-name>-app-openmetadata_pipeline_service_search_index
<project-name>-app-openmetadata_all <project-name>-app-openmetadata_query_search_index
<project-name>-app-openmetadata_all <project-name>-app-openmetadata_search_entity_search_index
<project-name>-app-openmetadata_all <project-name>-app-openmetadata_search_service_search_index
<project-name>-app-openmetadata_all <project-name>-app-openmetadata_storage_service_search_index
<project-name>-app-openmetadata_all <project-name>-app-openmetadata_stored_procedure_search_index
<project-name>-app-openmetadata_all <project-name>-app-openmetadata_table_search_index
<project-name>-app-openmetadata_all <project-name>-app-openmetadata_tag_search_index
<project-name>-app-openmetadata_all <project-name>-app-openmetadata_topic_search_index
<project-name>-app-openmetadata_chart <project-name>-app-openmetadata_chart_search_index
<project-name>-app-openmetadata_classification <project-name>-app-openmetadata_classification_search_index
<project-name>-app-openmetadata_classification <project-name>-app-openmetadata_tag_search_index
<project-name>-app-openmetadata_container <project-name>-app-openmetadata_container_search_index
<project-name>-app-openmetadata_dashboard <project-name>-app-openmetadata_chart_search_index
<project-name>-app-openmetadata_dashboard <project-name>-app-openmetadata_dashboard_search_index
<project-name>-app-openmetadata_dashboardDataModel <project-name>-app-openmetadata_dashboard_data_model_search_index
<project-name>-app-openmetadata_dashboardService <project-name>-app-openmetadata_chart_search_index
<project-name>-app-openmetadata_dashboardService <project-name>-app-openmetadata_dashboard_data_model_search_index
<project-name>-app-openmetadata_dashboardService <project-name>-app-openmetadata_dashboard_search_index
<project-name>-app-openmetadata_dashboardService <project-name>-app-openmetadata_dashboard_service_search_index
<project-name>-app-openmetadata_dataAsset <project-name>-app-openmetadata_chart_search_index
<project-name>-app-openmetadata_dataAsset <project-name>-app-openmetadata_container_search_index
<project-name>-app-openmetadata_dataAsset <project-name>-app-openmetadata_dashboard_data_model_search_index
<project-name>-app-openmetadata_dataAsset <project-name>-app-openmetadata_dashboard_search_index
<project-name>-app-openmetadata_dataAsset <project-name>-app-openmetadata_data_product_search_index
<project-name>-app-openmetadata_dataAsset <project-name>-app-openmetadata_database_schema_search_index
<project-name>-app-openmetadata_dataAsset <project-name>-app-openmetadata_database_search_index
<project-name>-app-openmetadata_dataAsset <project-name>-app-openmetadata_glossary_term_search_index
<project-name>-app-openmetadata_dataAsset <project-name>-app-openmetadata_ingestion_pipeline_search_index
<project-name>-app-openmetadata_dataAsset <project-name>-app-openmetadata_mlmodel_search_index
<project-name>-app-openmetadata_dataAsset <project-name>-app-openmetadata_pipeline_search_index
<project-name>-app-openmetadata_dataAsset <project-name>-app-openmetadata_search_entity_search_index
<project-name>-app-openmetadata_dataAsset <project-name>-app-openmetadata_stored_procedure_search_index
<project-name>-app-openmetadata_dataAsset <project-name>-app-openmetadata_table_search_index
<project-name>-app-openmetadata_dataAsset <project-name>-app-openmetadata_tag_search_index
<project-name>-app-openmetadata_dataAsset <project-name>-app-openmetadata_topic_search_index
<project-name>-app-openmetadata_dataProduct <project-name>-app-openmetadata_data_product_search_index
<project-name>-app-openmetadata_database <project-name>-app-openmetadata_database_schema_search_index
<project-name>-app-openmetadata_database <project-name>-app-openmetadata_database_search_index
<project-name>-app-openmetadata_database <project-name>-app-openmetadata_stored_procedure_search_index
<project-name>-app-openmetadata_database <project-name>-app-openmetadata_table_search_index
<project-name>-app-openmetadata_databaseSchema <project-name>-app-openmetadata_database_schema_search_index
<project-name>-app-openmetadata_databaseSchema <project-name>-app-openmetadata_stored_procedure_search_index
<project-name>-app-openmetadata_databaseSchema <project-name>-app-openmetadata_table_search_index
<project-name>-app-openmetadata_databaseService <project-name>-app-openmetadata_database_schema_search_index
<project-name>-app-openmetadata_databaseService <project-name>-app-openmetadata_database_search_index
<project-name>-app-openmetadata_databaseService <project-name>-app-openmetadata_database_service_search_index
<project-name>-app-openmetadata_databaseService <project-name>-app-openmetadata_stored_procedure_search_index
<project-name>-app-openmetadata_databaseService <project-name>-app-openmetadata_table_search_index
<project-name>-app-openmetadata_domain <project-name>-app-openmetadata_data_product_search_index
<project-name>-app-openmetadata_domain <project-name>-app-openmetadata_domain_search_index
<project-name>-app-openmetadata_entityReportData <project-name>-app-openmetadata_entity_report_data_index
<project-name>-app-openmetadata_glossary <project-name>-app-openmetadata_glossary_search_index
<project-name>-app-openmetadata_glossary <project-name>-app-openmetadata_glossary_term_search_index
<project-name>-app-openmetadata_glossaryTerm <project-name>-app-openmetadata_glossary_term_search_index
<project-name>-app-openmetadata_ingestionPipeline <project-name>-app-openmetadata_ingestion_pipeline_search_index
<project-name>-app-openmetadata_messagingService <project-name>-app-openmetadata_messaging_service_search_index
<project-name>-app-openmetadata_messagingService <project-name>-app-openmetadata_topic_search_index
<project-name>-app-openmetadata_metadataService <project-name>-app-openmetadata_metadata_service_search_index
<project-name>-app-openmetadata_mlModelService <project-name>-app-openmetadata_mlmodel_search_index
<project-name>-app-openmetadata_mlModelService <project-name>-app-openmetadata_mlmodel_service_search_index
<project-name>-app-openmetadata_mlmodel <project-name>-app-openmetadata_mlmodel_search_index
<project-name>-app-openmetadata_pipeline <project-name>-app-openmetadata_pipeline_search_index
<project-name>-app-openmetadata_pipelineService <project-name>-app-openmetadata_pipeline_search_index
<project-name>-app-openmetadata_pipelineService <project-name>-app-openmetadata_pipeline_service_search_index
<project-name>-app-openmetadata_query <project-name>-app-openmetadata_query_search_index
<project-name>-app-openmetadata_searchIndex <project-name>-app-openmetadata_search_entity_search_index
<project-name>-app-openmetadata_searchService <project-name>-app-openmetadata_search_entity_search_index
<project-name>-app-openmetadata_searchService <project-name>-app-openmetadata_search_service_search_index
<project-name>-app-openmetadata_storageService <project-name>-app-openmetadata_container_search_index
<project-name>-app-openmetadata_storageService <project-name>-app-openmetadata_storage_service_search_index
<project-name>-app-openmetadata_storedProcedure <project-name>-app-openmetadata_stored_procedure_search_index
<project-name>-app-openmetadata_table <project-name>-app-openmetadata_table_search_index
<project-name>-app-openmetadata_tag <project-name>-app-openmetadata_tag_search_index
<project-name>-app-openmetadata_team <project-name>-app-openmetadata_team_search_index
<project-name>-app-openmetadata_testCase <project-name>-app-openmetadata_test_case_search_index
<project-name>-app-openmetadata_testCaseResolutionStatus <project-name>-app-openmetadata_test_case_resolution_status_search_index
<project-name>-app-openmetadata_testSuite <project-name>-app-openmetadata_test_case_search_index
<project-name>-app-openmetadata_testSuite <project-name>-app-openmetadata_test_suite_search_index
<project-name>-app-openmetadata_topic <project-name>-app-openmetadata_topic_search_index
<project-name>-app-openmetadata_user <project-name>-app-openmetadata_user_search_index
<project-name>-app-openmetadata_webAnalyticEntityViewReportData <project-name>-app-openmetadata_web_analytic_entity_view_report_data_index
<project-name>-app-openmetadata_webAnalyticUserActivityReportData <project-name>-app-openmetadata_web_analytic_user_activity_report_data_index
Unfortunately, it tries to access the indices web_analytic_user_activity_report_data_index and entity_report_data_index without the prefix when I browse the UI and fails to do so (even if they exist in the index list with the prefixes):
An exception with message [Elasticsearch exception [type=security_exception, reason=action [indices:data/read/search] is unauthorized for user [zz-xyz] with effective roles [.build-view,.dev-view,.test-view,prj-zz-xyz] on indices [entity_report_data_index], this action is granted by the index privileges [read,all]]] was thrown while processing request.
An exception with message [Elasticsearch exception [type=security_exception, reason=action [indices:data/read/search] is unauthorized for user [zz-xyz] with effective roles [.build-view,.dev-view,.test-view,prj-zz-xyz] on indices [web_analytic_user_activity_report_data_index], this action is granted by the index privileges [read,all]]] was thrown while processing request.
Expected behavior
OpenMetadata uses indices, such as <project-name>-app-openmetadata_web_analytic_user_activity_report_data_index and <project-name>-app-openmetadata_entity_report_data_index with the prefixes included (I noticed it only with those two indices so far).
Version:
- OpenMetadata version: 1.4.1
Additional context We deploy OpenMetadata via the official Helm Chart.
hi @notniknot , did you try reindexing the application once ? You can do this by going to Settings /Applications/Search Indexing , go to configuration and set Recreate indexes to true , submit the setting and run reindexing
Hi @sonika-shah, thanks for the instructions. I ran the reindexing, but unfortunately, the error still persists.
Thanks for the information @notniknot , the full support for clusterAlias is available in version 1.5.0, which is currently in the pre-release stage. If you have an OpenMetadata instance set up in a testing environment, you might consider upgrading from version 1.4.1 to 1.5.0 and test it around.
Thanks. I will try out 1.5.0 and report back :)
Edit: @sonika-shah I got the same bug as https://github.com/open-metadata/OpenMetadata/issues/17593. Once that is resolved, I will try it out.
With 1.5.1, I do not encounter those errors anymore when I'm browsing the UI.