intake-esm icon indicating copy to clipboard operation
intake-esm copied to clipboard

Load only those columns which are defined in the catalog description

Open wachsylon opened this issue 3 years ago • 0 comments

Here is an idea if the underlying Dataframe is based on a catalog_file and in case it is opened and loaded with pandas read_csv.

Is your feature request related to a problem? Please describe. The catalog file of a catalog can be very large and contain many columns. It quickly fills user memory when the dataframe is loaded.

Describe the solution you'd like pandas read_csv allows to set kwarg usecols. One could assemble all columns from the description and set it as values. That reduces the memory requirements.

wachsylon avatar Mar 18 '22 13:03 wachsylon