open-metadata
API Endopoint([PUT] /api/v1/tags) not handle ElasticSearch error
Affected module API, Ingestion
Describe the bug openmetadata has an error when updating tags in metadata due ingestion.
$ metadata ingest -c templates/bigquery.yaml
+--------+---------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| From | Entity Name | Message | Stack Trace |
+========+=================================+==================================================================================================================================================================+===========================================================================================================================================================================+
| Sink | OMetaTagAndClassification [env] | Failed to ingest OMetaTagAndClassification [env] due to api request failure: 500 Server Error: Internal Server Error for url: http://localhost:39999/api/v1/tags | Traceback (most recent call last): |
| | | | File "/Users/user/.pyenv/versions/3.10.8/lib/python3.10/site-packages/metadata/ingestion/sink/metadata_rest.py", line 124, in _run |
| | | | return self._run_dispatch(record) |
| | | | File "/Users/user/.pyenv/versions/3.10.8/lib/python3.10/functools.py", line 926, in _method |
| | | | return method.__get__(obj, cls)(*args, **kwargs) |
| | | | File "/Users/user/.pyenv/versions/3.10.8/lib/python3.10/site-packages/metadata/ingestion/sink/metadata_rest.py", line 199, in write_classification_and_tag |
| | | | tag = self.metadata.create_or_update(record.tag_request) |
| | | | File "/Users/user/.pyenv/versions/3.10.8/lib/python3.10/site-packages/metadata/ingestion/ometa/ometa_api.py", line 275, in create_or_update |
| | | | resp = self.client.put( |
| | | | File "/Users/user/.pyenv/versions/3.10.8/lib/python3.10/site-packages/metadata/ingestion/ometa/client.py", line 288, in put |
| | | | return self._request("PUT", path, data) |
| | | | File "/Users/user/.pyenv/versions/3.10.8/lib/python3.10/site-packages/metadata/ingestion/ometa/client.py", line 189, in _request |
| | | | return self._one_request(method, url, opts, retry) |
| | | | File "/Users/user/.pyenv/versions/3.10.8/lib/python3.10/site-packages/metadata/ingestion/ometa/client.py", line 212, in _one_request |
| | | | resp.raise_for_status() |
| | | | File "/Users/user/.pyenv/versions/3.10.8/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status |
| | | | raise HTTPError(http_error_msg, response=self) |
| | | | requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://localhost:39999/api/v1/tags |
+--------+---------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
server log is following.
ERROR [2023-12-05 06:58:01,936] [I/O dispatcher 1] o.o.s.s.o.OpenSearchClient - Entity Update failed: OpenSearch exception [type=version_conflict_engine_exception, reason=[12383766-e65a-408f-a621-e848fe9ea897]: version conflict, required seqNo [30199], primary term [1]. current document has seqNo [30202] and primary term [1]]
To Reproduce It is not clear what causes the error, as some sources generate the error and others do not.
Expected behavior
The ingestion command and api need an option to ignore version conflict errors.
conflicts=proceed
https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update-by-query.html
Version:
- Python version: 3.10.
- OpenMetadata version: 1.2.0
- OpenMetadata Ingestion package version:
openmetadata-ingestion[all]==1.2.0.1
Additional context Add any other context about the problem here.
@TrsNium we do not throw error 503 if its unable to update the OpenSearch. Need more logs fix this , please see if there are any error stack traces in server
we are not able to reproduce the issue in latest version, you can reopen the issue, if you face similar concern
