Matthew Lennie

icon indicating a website link https://www.researchgate.net/profile/Matthew_Lennie

icon location Wind energy and data science researcher. My github only contains open code projects and coding practice and doesn't represent my entire portfolio.

Results 9 comments of Matthew Lennie

If opening with xarray open_mfdataset and parallel=True it will fail unless you have previously opened it with parallel=False

> Minimal repro: > > ``` > import xarray as xr > ds = xr.open_mfdataset('gfs.0p25.201511*00.f0*.grib2', engine='cfgrib', combine='nested', concat_dim=['step'], parallel=True, chunks=24, backend_kwargs={'filter_by_keys': {'typeOfLevel': 'surface'}, 'indexpath': ''}) > ``` > > Expected...

If opening with xarray open_mfdataset and parallel=True it will fail unless you have previously opened it with parallel=False

That's interesting. Do you happen to have a theory of why this error would appear in parallel but not in serial? On 06.10.2020 09:09, Guido Cioni wrote: > I can...

If opening with xarray open_mfdataset and parallel=True it will fail unless you have previously opened it with parallel=False

Could it be that eccodes isn't thread safe some how? It seems that when manually open multiple files using CFgrib.open_datasets via multiple processes I don't get the error. I do...

If opening with xarray open_mfdataset and parallel=True it will fail unless you have previously opened it with parallel=False

Thanks for the information. I am stumped then, can you think of another reason why I (and others) would see this behavior? m On 08.10.2020 14:47, shahramn wrote: > The...

If opening with xarray open_mfdataset and parallel=True it will fail unless you have previously opened it with parallel=False

Awesome. Thanks for looking into it. Not all heroes wear capes :) On 08.10.2020 15:01, shahramn wrote: > Looks like the conda recipe does NOT enable the thread safety flags....

If opening with xarray open_mfdataset and parallel=True it will fail unless you have previously opened it with parallel=False

I can also confirm. I just ran a test using delays = [] for file in files: delays.append(dask.delay(cfgrib.opendatasets(file), backend_kwargs={"indexpath":""})) client.persist(delays) It previous resulted in killed workers as described. Now the...

Performance of GRIB files read with dask and dask.distributed can be improved.

Hi @alexamici I would just like to add a data point regarding performance. I am trying to open a 320mb file on a HPC system where bandwidth is not an...

Lazy fitting

Hey all, IMO, lazy fit will be a fantastic addition. I had some real pain using the StandardScaler with the column transformer because it called compute for each column. I...

Lazy fitting

Further to my previous comment. A rough idea of a Column transformer implementation could be done thusly. This would just work with data in the form of dataframes, but that...