astro-sdk
astro-sdk copied to clipboard
Investigate - 10MB file transfer from gcs to redshift is taking too much time
Describe the bug Currently, the data transfer from GCS to Redshift is taking too much time. For a 100KB file time taken was 6min For the 10Mb file Process was getting killed.
Version
- Astro: [1.0.2]
To Reproduce Steps to reproduce the behavior:
- Update the conf.json file with the datasets of size 10MB/100kb and database to redshift
- run
run.sh
in terminal
Expected behavior Data should load in a reasonable time and the process shouldn't be killed.