Investigate astro-sdk memory issue

[pankajastro]

trafficstars

Describe the bug Earlier we were using n2-standard-4 type cluster that we had to change n2-standard-8 while doing benchmarking for bigger files. We also increased our benchmark pod size to 16GI from 6GI which is a bit concerning.

We can also check in GKE UI once the benchmark pod requested 17.18 GB of memory. Checking memory_full_info_rss in bigquery benchmark table might be helfull.

Acceptance Criteria

• [ ] Add limit for postgress pod
• [ ] Profile why and when astro-sdk consumes so much memory

Version

• Astro: [e.g. 0.6.0]
• OS: [eg. Debian]

To Reproduce Steps to reproduce the behavior:

1. Write the DAG '...'
2. Create connection '....'
3. Run using '....'
4. See error

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Additional context Add any other context about the problem here.