astro-sdk icon indicating copy to clipboard operation
astro-sdk copied to clipboard

Published 20 hours ago • verified publisher icon astronomer

Duckdb not working for files larger than 250 MB

Open sunank200 opened this issue 2 years ago • 0 comments

Describe the bug A clear and concise description of what the bug is. Trying to load a CSV file from AWS into a DuckDB table. The process keeps filling up the container memory until the machine I'm running it on crashes.

Issue reported on this thread: https://astronomer.slack.com/archives/C02B8SPT93K/p1681205784093889

Expected behavior

  • Debug the cause
  • File should be loaded as directly with directly with the python client

sunank200 avatar Apr 11 '23 10:04 sunank200