intake-esm
intake-esm copied to clipboard
Load only those columns which are defined in the catalog description
Here is an idea if the underlying Dataframe is based on a catalog_file and in case it is opened and loaded with pandas read_csv.
Is your feature request related to a problem? Please describe. The catalog file of a catalog can be very large and contain many columns. It quickly fills user memory when the dataframe is loaded.
Describe the solution you'd like
pandas read_csv allows to set kwarg usecols. One could assemble all columns from the description and set it as values. That reduces the memory requirements.