astro-sdk icon indicating copy to clipboard operation
astro-sdk copied to clipboard

Published 20 hours ago • verified publisher icon astronomer

Investigate - 10MB file transfer from gcs to redshift is taking too much time

Open utkarsharma2 opened this issue 1 year ago • 1 comments

Describe the bug Currently, the data transfer from GCS to Redshift is taking too much time. For a 100KB file time taken was 6min For the 10Mb file Process was getting killed.

Version

  • Astro: [1.0.2]

To Reproduce Steps to reproduce the behavior:

  1. Update the conf.json file with the datasets of size 10MB/100kb and database to redshift
  2. run run.sh in terminal

Expected behavior Data should load in a reasonable time and the process shouldn't be killed.

utkarsharma2 avatar Sep 08 '22 04:09 utkarsharma2