pyam icon indicating copy to clipboard operation
pyam copied to clipboard

pandas over-ambitiously casts to types when reading from file

Open danielhuppmann opened this issue 3 years ago • 0 comments

When reading a data file, pandas tries to identify the most suitable type for each column. This can result in having types other than string in the index columns, which can cause problems in subsequent processing workflows.

An example: if you have an entry 2000-12-01 00:00:00 in the region column, this is cast to datetime, which causes the subsequent validation step to fail - see the related issue at https://github.com/openENTRANCE/nomenclature/issues/89.

Even if the non-string entries may be wrong or nonsensical, casting all columns other than the time domain to string could increase stability of the behavior in downstream workflows.

danielhuppmann avatar Jan 27 '21 13:01 danielhuppmann