cdsapi icon indicating copy to clipboard operation
cdsapi copied to clipboard

Load into memory without saving to disk

Open lgloege opened this issue 4 years ago • 1 comments

Is there any way to load the requested data into memory without saving to disk? I am trying to load the data as netcdf and do something like this:

import cdsapi

c = cdsapi.Client()

params =  {
    'product_type': 'reanalysis',
    'variable': 'total_precipitation',
    'year': '2019',
    'month': '01',
    'day': '01',
    'time': '00:00',
    'format': 'netcdf',
    'grid':[1.0, 1.0],
     } 

data = c.retrieve('reanalysis-era5-single-levels',  params)
ds = data.load() # doesn't exist, but could load data into memory

# process the data below here

I know .download() will save to disk. but I am curious if there is any way to implement something like a .load() method which would load the data into memory as an xarray dataset. This way one could load the data and process it in one script. The current solution is to download the data and just throw away the original dataset after processing, but I feel like this process could be streamlined.

I looked at the download method but couldn't figure out a way to implement some type of load method. Any thoughts?

Thanks for making this package. This has saved me so much time!

lgloege avatar Oct 01 '20 15:10 lgloege

I figured out a solution using urlopen. Here is a small working example

import cdsapi
import xarray as xr
from urllib.request import urlopen

c = cdsapi.Client()

# API request
params = {'format': 'netcdf',
        'product_type': 'monthly_averaged_reanalysis',
        'variable': '2m_temperature',
        'year': list(map(str, range(2000, 2022))),
        'month': list(map("{:02d}".format, range(1,13))),
        'time': '00:00',
        'grid': [1.0, 1.0],}

# retrieve the location of the file
fl = c.retrieve('reanalysis-era5-single-levels-monthly-means', params) 

# load into memory 
with urlopen(fl.location) as f:
    ds = xr.open_dataset(f.read())

lgloege avatar Mar 27 '21 04:03 lgloege

Hi, Thanks @lgloege for providing a solution. Eddy

EddyCMWF avatar Feb 28 '23 09:02 EddyCMWF