error while loading read_mds hFac llc file

[apatlpo]

I am trying to update the following code with the xmitgcm (as of 6147c27f1254042c51b7d190fcbe4e83c0ff380c ):

https://github.com/apatlpo/mit_equinox/blob/f1b0dd2fa6d129c447a81240134df80d57960b19/mitequinox/utils.py#L89-L147

I have updated lines 128-129 with:

    hfac = xm.utils.read_mds(grid_dir + 'hFac' + point,
                            use_mmap=True, use_dask=False, extra_metadata=llc4320)

where llc4320 = xm.utils.get_extra_metadata(domain='llc', nx=4320) but it fails with the following error:

--------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-9-71a191386415> in <module>()
     22         p = 'S'
     23     #
---> 24     ds = get_compressed_data(v, data_dir, grid_dir, iters=iters, time=time, client=client, point=p)
     25     #
     26     #tmp

~/mit_equinox/mitequinox/utils.py in get_compressed_data(varname, data_dir, grid_dir, ds_index, ds, iters, time, client, k, point, **kwargs)
    128     # load mask from raw data
    129     hfac = xm.utils.read_mds(grid_dir + 'hFac' + point,
--> 130                             use_mmap=True, use_dask=False, extra_metadata=llc4320)
    131     #                        use_mmap=True, use_dask=False, force_dict=False)
    132     mask = hfac[k]>0

~/xmitgcm/xmitgcm/utils.py in read_mds(fname, iternum, use_mmap, endian, shape, dtype, use_dask, extra_metadata, chunks, llc, llc_method, legacy)
    264     d = read_all_variables(file_metadata['fldList'], file_metadata,
    265                            use_mmap=use_mmap, use_dask=use_dask,
--> 266                            chunks=chunks)
    267 
    268     # convert list into dictionary

~/xmitgcm/xmitgcm/utils.py in read_all_variables(variable_list, file_metadata, use_mmap, use_dask, chunks)
    671         elif chunks == "3D":
    672             out.append(read_3D_chunks(variable, file_metadata,
--> 673                                       use_mmap=use_mmap, use_dask=use_dask))
    674 
    675     return out

~/xmitgcm/xmitgcm/utils.py in read_3D_chunks(variable, file_metadata, use_mmap, use_dask)
    797 
    798     if not use_dask:
--> 799         data = data.compute()
    800 
    801     return data
...
~/xmitgcm/xmitgcm/utils.py in load_chunk()
    783         return _read_xyz_chunk(variable, file_metadata,
    784                                rec=rec,
--> 785                                use_mmap=use_mmap)[None]
    786 
    787     chunks = (1, file_metadata['nz'], file_metadata['ny'], file_metadata['nx'])

~/xmitgcm/xmitgcm/utils.py in _read_xyz_chunk()
    825     if file_metadata['has_faces'] and ((file_metadata['nx'] > 1) or
    826                                        (file_metadata['ny'] > 1)):
--> 827         raise ValueError("_read_xyz_chunk cannot be called with llc type grid")
    828 
    829     # size of the data element

ValueError: _read_xyz_chunk cannot be called with llc type grid

Any advice on this?

[raphaeldussin]

Hi aurélien,

the idea is that llc configurations are only using 2d (x,y) chunks. The args of the function is :

def read_mds(fname, iternum=None, use_mmap=True, endian='>', shape=None, dtype=None, use_dask=True, extra_metadata=None, chunks="3D", llc=False, llc_method="smallchunks", legacy=True):

you can either add chunks="2D" for your llc configurations or set the llc arg to True (sets chunks to 2D in the code).

[rabernat]

Although it is technically part of the public API, I’m curious why you are using read_mds rather than the higher level open_mdsdataset? I consider this a semi private function.

Sent from my iPhone

On Sep 25, 2018, at 8:13 AM, raphael dussin [email protected] wrote:

Hi aurélien,

the idea is that llc configurations are only using 2d (x,y) chunks. The args of the function is :

def read_mds(fname, iternum=None, use_mmap=True, endian='>', shape=None, dtype=None, use_dask=True, extra_metadata=None, chunks="3D", llc=False, llc_method="smallchunks", legacy=True):

you can either add chunks="2D" for your llc configurations or set the llc arg to True (sets chunks to 2D in the code).

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[apatlpo]

I can return the question to you Ryan, as you wrote this piece of code ;)

[apatlpo]

as a reminder the goal is to read compressed ll4320 data and mask it

[apatlpo]

Should I understand that I can do that with open_mdsdataset ?

[rabernat]

Of course...you are working with the compressed data. I had overlooked that.

We should incorporate that module into xmitgcm proper.

Sent from my iPhone

On Sep 25, 2018, at 8:29 AM, Aurélien Ponte [email protected] wrote:

Should I understand that I can do that with open_mdsdataset ?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[apatlpo]

let me know if I can help with anything here or if you'd rather deal with this