gdrivefs
gdrivefs copied to clipboard
Writes to Zarr using gdrivefs extremely slow
I am writing a Zarr file to Google Drive using gdrivefs and Xarray, and the writes are extremely slow. The file is local to my notebook and is about 1.5 GB in size. It has a lot of variables (89), and it is chunked by day into 365 days. It has been working for about an hour and is approximately 10% finished. Here are the steps I am using:
ds = xr.open_zarr('ICE.2000.01-12.c41.zarr', consolidated=True, decode_times=False, chunks=False,
decode_cf=False, mask_and_scale=True)
gdfs = gdrivefs.GoogleDriveFileSystem(root_file_id='1PCBDhk5f3v5PoPCY3Rdcqgy4S_Yj2kCC', token='cache')
mapper = gdfs.get_mapper('ICE.2000.01-12.c41.zarr')
ds.to_zarr(mapper, compute=True, consolidated=True, encoding={})
Anything obvious that I am doing wrong here? Thanks.
cc: @raf-antwerpen