pgsync icon indicating copy to clipboard operation
pgsync copied to clipboard

getting 429 while sending bulk request

Open ChiragGangwani opened this issue 2 years ago • 1 comments

PGSync version: 2.5.1

Postgres version: 12

Elasticsearch version: latest

Redis version: latest

Python version: 3.10

Problem Description: While updating 9000 row in db some time it post to open search but some time it through an exception of bulk request.

we have small instance. these parameter are set

ELASTICSEARCH_CHUNK_SIZE=1000 ELASTICSEARCH_MAX_CHUNK_BYTES=3242880 ELASTICSEARCH_MAX_RETRIES=14 ELASTICSEARCH_QUEUE_SIZE=1 ELASTICSEARCH_STREAMING_BULK=True ELASTICSEARCH_THREAD_COUNT=1 ELASTICSEARCH_TIMEOUT=320 QUERY_CHUNK_SIZE=500

Error Message (if any):

ERROR:pgsync.search_client: Exception TransportError(429, '429 Too Many Requests /_bulk')
data1-pgsync-1  | Traceback (most recent call last):

ChiragGangwani avatar Aug 07 '23 11:08 ChiragGangwani