"OSError: Permission denied" when creating a file in a directory that doesn't exist

[pelson]

When truing to create a dataset in a directory that doesn't exist, a "Permission error" ensues:

>>> import netCDF4
>>> ds = netCDF4.Dataset("path/to/somewhere/imaginary/test.nc", "w")
HDF5-DIAG: Error detected in HDF5 (1.8.17) thread 0:
  #000: H5F.c line 522 in H5Fcreate(): unable to create file
    major: File accessibilty
    minor: Unable to open file
  #001: H5Fint.c line 1003 in H5F_open(): unable to open file: time = Wed Jan  3 15:41:03 2018
, name = 'path/to/somewhere/imaginary/test.nc', tent_flags = 13
    major: File accessibilty
    minor: Unable to open file
  #002: H5FD.c line 993 in H5FD_open(): open failed
    major: Virtual File Layer
    minor: Unable to initialize object
  #003: H5FDsec2.c line 339 in H5FD_sec2_open(): unable to open file: name = 'path/to/somewhere/imaginary/test.nc', errno = 2, error message = 'No such file or directory', flags = 13, o_flags = 602
    major: File accessibilty
    minor: Unable to open file
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "netCDF4/_netCDF4.pyx", line 1848, in netCDF4._netCDF4.Dataset.__init__ (netCDF4/_netCDF4.c:13981)
IOError: Permission denied

It would be helpful if the message gave more detail. For example: "Cannot create the dataset as path/to/somewhere/imaginary does not exist".

I couldn't find the bit of code that raises the "Permission denied" error in the codebase (nor in netcdf-c), so just wanted to raise this to see if there is a place that this kind of handling belongs?

I'm more than happy for this issue to be rejected if "Permission denied" is considered good enough an error message.

Cheers,

(P.S. I also notice that accessibilty is misspelt in the stderr, but assume that is in the HDF5 library?)

[jswhit]

That error message is returned by nc_strerror, which in turn is forwarding the system error using strerror.

I agree a more informative message would be nice, but I don't immediately see how we would catch the error.

[pelson]

Thanks for the quick response. I was doing a bit of digging thanks to your comment, and I'm just trying to get to the bottom of where the "Permission denied" message is coming from. Do you know if the actual implementation of nc_strerror is the one at https://github.com/Unidata/netcdf-c/blob/master/libdispatch/derror.c#L86?

This report has bubbled down from Iris. I'm personally in favour of trying to improve the message, and ideally it would be great if everybody could benefit from the improved message (not just Iris users). I guess even better would be for this to go into netcdf-c, so that it isn't just Python users who benefit.

[shoyer]

Perhaps the fix in https://github.com/Unidata/netcdf4-python/pull/727 would be relevant here?

[jswhit]

Yes, I believe the message is coming from libdispatch/derror.c, but @DennisHeimbigner would know for sure.

[DennisHeimbigner]

It looks like it is coming from derror.c The problem is that netcdf-c errors are reported as a single integer (not unlike the std linux errno errors). As such, it carries only a limited amount of information. So it is not possible to distinguish, for example, missing file from missing directory from bad permissions.