pgsync icon indicating copy to clipboard operation
pgsync copied to clipboard

OpenSearch returning 400

Open leokaynan-bayer opened this issue 2 years ago • 1 comments

PGSync version: 2.5.0

Postgres version: aurora postgresql 12

Elasticsearch version: OpenSearch 2.7 (latest)

Redis version:

Python version:

Problem Description: OpenSearch is returning status code 400.

The connection with opensearch and postgresql is working perfectly.

it is possible to make requests via CURL.

When running the bootstrap and pgsync commands the index is created but not populated.

image

2023-08-22 15:03:18.941:ERROR:pgsync.search_client: Exception RequestError(400, 'illegal_argument_exception', 'Action/metadata line [1] contains an unknown parameter [_type]') Traceback (most recent call last): File "/home/ubuntu/.local/lib/python3.10/site-packages/pgsync/search_client.py", line 133, in bulk self._bulk( File "/home/ubuntu/.local/lib/python3.10/site-packages/pgsync/search_client.py", line 171, in _bulk for _ in self.streaming_bulk( File "/home/ubuntu/.local/lib/python3.10/site-packages/elasticsearch/helpers/actions.py", line 329, in streaming_bulk for data, (ok, info) in zip( File "/home/ubuntu/.local/lib/python3.10/site-packages/elasticsearch/helpers/actions.py", line 256, in _process_bulk_chunk for item in gen: File "/home/ubuntu/.local/lib/python3.10/site-packages/elasticsearch/helpers/actions.py", line 195, in _process_bulk_chunk_error raise error File "/home/ubuntu/.local/lib/python3.10/site-packages/elasticsearch/helpers/actions.py", line 240, in _process_bulk_chunk resp = client.bulk("\n".join(bulk_actions) + "\n", *args, **kwargs) File "/home/ubuntu/.local/lib/python3.10/site-packages/elasticsearch/client/utils.py", line 168, in _wrapped return func(*args, params=params, headers=headers, **kwargs) File "/home/ubuntu/.local/lib/python3.10/site-packages/elasticsearch/client/init.py", line 463, in bulk return self.transport.perform_request( File "/home/ubuntu/.local/lib/python3.10/site-packages/elasticsearch/transport.py", line 415, in perform_request raise e File "/home/ubuntu/.local/lib/python3.10/site-packages/elasticsearch/transport.py", line 381, in perform_request status, headers_response, data = connection.perform_request( File "/home/ubuntu/.local/lib/python3.10/site-packages/elasticsearch/connection/http_urllib3.py", line 277, in perform_request self._raise_error(response.status, raw_data) File "/home/ubuntu/.local/lib/python3.10/site-packages/elasticsearch/connection/base.py", line 330, in _raise_error raise HTTP_EXCEPTIONS.get(status_code, TransportError)( elasticsearch.exceptions.RequestError: RequestError(400, 'illegal_argument_exception', 'Action/metadata line [1] contains an unknown parameter [_type]') 0:00:41.123086 (41.12 sec) Traceback (most recent call last): File "/home/ubuntu/.local/bin/pgsync", line 7, in sync.main() File "/home/ubuntu/.local/lib/python3.10/site-packages/click/core.py", line 1130, in call return self.main(*args, **kwargs) File "/home/ubuntu/.local/lib/python3.10/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/home/ubuntu/.local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/ubuntu/.local/lib/python3.10/site-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) File "/home/ubuntu/.local/lib/python3.10/site-packages/pgsync/sync.py", line 1451, in main sync.pull() File "/home/ubuntu/.local/lib/python3.10/site-packages/pgsync/sync.py", line 1222, in pull self.search_client.bulk( File "/home/ubuntu/.local/lib/python3.10/site-packages/pgsync/search_client.py", line 133, in bulk self._bulk( File "/home/ubuntu/.local/lib/python3.10/site-packages/pgsync/search_client.py", line 171, in _bulk for _ in self.streaming_bulk( File "/home/ubuntu/.local/lib/python3.10/site-packages/elasticsearch/helpers/actions.py", line 329, in streaming_bulk for data, (ok, info) in zip( File "/home/ubuntu/.local/lib/python3.10/site-packages/elasticsearch/helpers/actions.py", line 256, in _process_bulk_chunk for item in gen: File "/home/ubuntu/.local/lib/python3.10/site-packages/elasticsearch/helpers/actions.py", line 195, in _process_bulk_chunk_error raise error File "/home/ubuntu/.local/lib/python3.10/site-packages/elasticsearch/helpers/actions.py", line 240, in _process_bulk_chunk resp = client.bulk("\n".join(bulk_actions) + "\n", *args, **kwargs) File "/home/ubuntu/.local/lib/python3.10/site-packages/elasticsearch/client/utils.py", line 168, in _wrapped return func(*args, params=params, headers=headers, **kwargs) File "/home/ubuntu/.local/lib/python3.10/site-packages/elasticsearch/client/init.py", line 463, in bulk return self.transport.perform_request( File "/home/ubuntu/.local/lib/python3.10/site-packages/elasticsearch/transport.py", line 415, in perform_request raise e File "/home/ubuntu/.local/lib/python3.10/site-packages/elasticsearch/transport.py", line 381, in perform_request status, headers_response, data = connection.perform_request( File "/home/ubuntu/.local/lib/python3.10/site-packages/elasticsearch/connection/http_urllib3.py", line 277, in perform_request self._raise_error(response.status, raw_data) File "/home/ubuntu/.local/lib/python3.10/site-packages/elasticsearch/connection/base.py", line 330, in _raise_error raise HTTP_EXCEPTIONS.get(status_code, TransportError)( elasticsearch.exceptions.RequestError: RequestError(400, 'illegal_argument_exception', 'Action/metadata line [1] contains an unknown parameter [_type]')

https://opensearch.org/docs/1.2/opensearch/rest-api/document-apis/bulk/ image

leokaynan-bayer avatar Aug 22 '23 14:08 leokaynan-bayer

Try running it with export ELASTICSEARCH=False.

Denton-L avatar Aug 25 '23 11:08 Denton-L