airbyte icon indicating copy to clipboard operation
airbyte copied to clipboard

Published 20 hours ago • verified publisher icon airbytehq

[destination-redshift] connector memory bug

Open Auric-Manteo opened this issue 1 year ago • 1 comments

Connector Name

destination-redshift

Connector Version

0.6.8

What step the error happened?

During the sync

Relevant information

Some memory issue - see the screenshot for details. The sync is MySQL using CDC with the latest version of all connectors as well as Airbyte on Kubernetes / helm. The sync seems to load data forever and does not stop by itself (12h+ 200GB+ counting, while the tables are about 100GB all together). Kubernetes has 25GB RAM and 8 CPUs. "Only" 10GB RAM seem to be used tops.

Relevant log output

image

Contribute

  • [ ] Yes, I want to contribute

Auric-Manteo avatar Oct 22 '23 14:10 Auric-Manteo

Does this problem happen on every sync, or is it sporadic? Do you have any streams with particularly large records you are syncing?

evantahler avatar May 09 '24 16:05 evantahler

this is a very old connector. Can we close this and recreate it if it happens again?

stephane-airbyte avatar May 21 '24 20:05 stephane-airbyte

yes, go ahead I don't know if it's solved. We never got CDC to work reliably, but in case we try again and we still run into the same error I can reopen the issue.

Auric-Manteo avatar May 22 '24 05:05 Auric-Manteo