gdrivefs icon indicating copy to clipboard operation
gdrivefs copied to clipboard

Writes to Zarr using gdrivefs extremely slow

Open tjcrone opened this issue 4 years ago • 0 comments

I am writing a Zarr file to Google Drive using gdrivefs and Xarray, and the writes are extremely slow. The file is local to my notebook and is about 1.5 GB in size. It has a lot of variables (89), and it is chunked by day into 365 days. It has been working for about an hour and is approximately 10% finished. Here are the steps I am using:

ds = xr.open_zarr('ICE.2000.01-12.c41.zarr', consolidated=True, decode_times=False, chunks=False,
                  decode_cf=False, mask_and_scale=True)
gdfs = gdrivefs.GoogleDriveFileSystem(root_file_id='1PCBDhk5f3v5PoPCY3Rdcqgy4S_Yj2kCC', token='cache')
mapper = gdfs.get_mapper('ICE.2000.01-12.c41.zarr')
ds.to_zarr(mapper, compute=True, consolidated=True, encoding={})

Anything obvious that I am doing wrong here? Thanks.

cc: @raf-antwerpen

tjcrone avatar Dec 04 '20 00:12 tjcrone