intake-esm icon indicating copy to clipboard operation
intake-esm copied to clipboard

Concatenate datasets when some variables are present in one dataset and not present in other dataset

Open andersy005 opened this issue 4 years ago • 3 comments

Currently, xarray.concat() doesn't know how to deal with datasets for which one dataset contains some variables and these same variables are not present in the rest of the datasets to concatenate.

To address this issue, there are two options:

  • Add NaNs by determining the set of variables that are not in all datasets but at least in one dataset and use .full_like(..., fill_value=np.nan) to add the variables not present in the other dataset (
  • Dropping the problematic variables Either option would probably need to be done prior calling xr.concat() in join_new() and join_existing() introduced in #135

andersy005 avatar Oct 15 '19 19:10 andersy005

Upstream issue: https://github.com/pydata/xarray/issues/508

This would be a relatively easy fix in concat if you're up for it. I'm happy to help.

dcherian avatar Oct 15 '19 21:10 dcherian

Sure. I will try working on this during the CMIp6 hackathon

andersy005 avatar Oct 15 '19 21:10 andersy005

See https://github.com/pydata/xarray/issues/3545. It could use some real world testing if you have the time...

dcherian avatar Nov 18 '19 00:11 dcherian