cf.read() not working for .pp files

[am-thyst]

Just freshly downloaded cf-python using conda:

$ conda install -c ncas -c conda-forge cf-python cf-plot udunits2
$ conda install -c conda-forge mpich esmpy

but for some reason I can't open any .pp files. I tried with the tutorial .pp file:

import cf
pp = cf.read('umfile.pp')

traceback:

Traceback (most recent call last):
  File "~/miniconda3/envs/workenv/lib/python3.8/site-packages/cf/read_write/read.py", line 723, in read
    ftype = file_type(filename)
  File "~/miniconda3/envs/workenv/lib/python3.8/site-packages/cf/read_write/read.py", line 1100, in file_type
    raise IOError(f"Can't determine format of file {filename}")
OSError: Can't determine format of file umfile.pp

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "~/miniconda3/envs/workenv/lib/python3.8/site-packages/cfdm/decorators.py", line 171, in verbose_override_wrapper
    return method_with_verbose_kwarg(*args, **kwargs)
  File "~/miniconda3/envs/workenv/lib/python3.8/site-packages/cf/read_write/read.py", line 741, in read
    raise ValueError(message)
ValueError: Can't determine format of file umfile.pp

So I tried adding the format: pp = cf.read('umfile.pp',um={'fmt': 'PP'})

and got the following traceback:

Traceback (most recent call last):
  File "~/miniconda3/envs/workenv/lib/python3.8/site-packages/cf/data/functions.py", line 128, in _open_um_file
    f = File(
  File "~/miniconda3/envs/workenv/lib/python3.8/site-packages/cf/umread_lib/umfile.py", line 55, in __init__
    c = cInterface.CInterface()
  File "~/miniconda3/envs/workenv/lib/python3.8/site-packages/cf/umread_lib/cInterface.py", line 136, in __init__
    self.lib = CT.CDLL(lib_path)
  File "~/miniconda3/envs/workenv/lib/python3.8/ctypes/__init__.py", line 381, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: ~/miniconda3/envs/workenv/lib/python3.8/site-packages/cf/umread_lib/c-lib/umfile.so: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "~/miniconda3/envs/workenv/lib/python3.8/site-packages/cfdm/decorators.py", line 171, in verbose_override_wrapper
    return method_with_verbose_kwarg(*args, **kwargs)
  File "~/miniconda3/envs/workenv/lib/python3.8/site-packages/cf/read_write/read.py", line 757, in read
    file_contents = _read_a_file(
  File "~/miniconda3/envs/workenv/lib/python3.8/site-packages/cfdm/decorators.py", line 171, in verbose_override_wrapper
    return method_with_verbose_kwarg(*args, **kwargs)
  File "~/miniconda3/envs/workenv/lib/python3.8/site-packages/cf/read_write/read.py", line 1032, in _read_a_file
    out = UM.read(
  File "~/miniconda3/envs/workenv/lib/python3.8/site-packages/cfdm/decorators.py", line 171, in verbose_override_wrapper
    return method_with_verbose_kwarg(*args, **kwargs)
  File "~/miniconda3/envs/workenv/lib/python3.8/site-packages/cf/read_write/um/umread.py", line 3503, in read
    f = self.file_open(filename)
  File "~/miniconda3/envs/workenv/lib/python3.8/site-packages/cf/read_write/um/umread.py", line 3586, in file_open
    return _open_um_file(
  File "~/miniconda3/envs/workenv/lib/python3.8/site-packages/cf/data/functions.py", line 140, in _open_um_file
    raise Exception(error)
Exception: ~/miniconda3/envs/workenv/lib/python3.8/site-packages/cf/umread_lib/c-lib/umfile.so: cannot open shared object file: No such file or directory

version info: Platform: Linux-3.10.0-957.27.2.el7.x86_64-x86_64-with-glibc2.10 HDF5 library: 1.12.1 netcdf library: 4.8.1 udunits2 library: ~/miniconda3/envs/workenv/lib/libudunits2.so.0 Python: 3.8.5 netCDF4: 1.5.8 cftime: 1.6.0 numpy: 1.19.2 psutil: 5.7.2 scipy: 1.8.0 matplotlib: 3.2.2 ESMF: 8.2.0 cfdm: 1.9.0.2 cfunits: 3.3.4 cfplot: 3.1.18 cf: 3.12.0

[sadielbartholomew]

Thanks for raising this @am-thyst and for providing so much detail, which is ideal to help us work out what is going on.

@davidhassell as you may also recognise, this looks like the traceback that occurs when the C library under cf/umread_lib/c-lib hasn't been built with make, so possibly that didn't get built as it should in this round of packaging?

[davidhassell]

Yes - for some reason the binary that is used for reading PP files didn't get built during the conda install - we'll look into that. However, there is a quick fix that will allow you to proceed - simply to run make in the appropriate directory:

$ cd ~/miniconda3/envs/workenv/lib/python3.8/site-packages/cf/umread_lib/c-lib
$ make

[am-thyst]

Thanks @davidhassell , that's solved the problem :)

[vichawans]

I also just downloaded cf-python and also did the previous bug fix (make in c-lib directory) but another error occurs.

cf.read(pp_dir,fmt='UM', um={'fmt':'PP'})

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_47855/3596004371.py in <module>
----> 1 cf.read(pp_dir, um={'fmt':'PP'})

~/.conda/envs/cmip6-env/lib/python3.9/site-packages/cfdm/decorators.py in verbose_override_wrapper(*args, **kwargs)
    169         # enabling
    170         try:
--> 171             return method_with_verbose_kwarg(*args, **kwargs)
    172         except Exception:
    173             raise

~/.conda/envs/cmip6-env/lib/python3.9/site-packages/cf/read_write/read.py in read(files, external, verbose, warnings, ignore_read_error, aggregate, nfields, squeeze, unsqueeze, fmt, cdl_string, select, extra, recursive, followlinks, um, chunk, field, height_at_top_of_model, select_options, follow_symlinks, mask, warn_valid, domain)
    755             # Read the file
    756             # --------------------------------------------------------
--> 757             file_contents = _read_a_file(
    758                 filename,
    759                 ftype=ftype,

~/.conda/envs/cmip6-env/lib/python3.9/site-packages/cfdm/decorators.py in verbose_override_wrapper(*args, **kwargs)
    169         # enabling
    170         try:
--> 171             return method_with_verbose_kwarg(*args, **kwargs)
    172         except Exception:
    173             raise

~/.conda/envs/cmip6-env/lib/python3.9/site-packages/cf/read_write/read.py in _read_a_file(filename, ftype, aggregate, aggregate_options, ignore_read_error, verbose, warnings, external, selected_fmt, um, extra, height_at_top_of_model, chunk, mask, warn_valid, select, domain)

[davidhassell]

Hi - quick suggestion, drop the um={'fmt':'PP'} keyword. It shouldn't be necessary, and has a bug in it which has been fixed for the next version (3.13.0), but hasn't yet been released. See https://github.com/NCAS-CMS/cf-python/issues/306.

Thanks, David

[sadielbartholomew]

Hi @vichawans, did David's suggestion above solve your problem? Just asking so I can close this Issue if it is no longer relevant. Thanks.

[davidhassell]

Closing now - please reopen if there are further questions/problems.