astro-sdk icon indicating copy to clipboard operation
astro-sdk copied to clipboard

load_file() - Complete Parquet file is loaded in Memory

Open utkarsharma2 opened this issue 3 years ago • 0 comments

Describe the bug When inferring schema complete Parquet is loaded in memory. https://github.com/astronomer/astro-sdk/blob/e4111397387e43ba7c8d2c0f5645376c466d271a/python-sdk/src/astro/files/types/parquet.py#L45

Version

  • Astro: 1.1.1

Expected behavior Research - if there is a better way to read schema information from parquet instead of not loading them in memory.

Additional context Add any other context about the problem here.

utkarsharma2 avatar Oct 14 '22 07:10 utkarsharma2