netcdf4-python
netcdf4-python copied to clipboard
Unidata
DLL installation broken on Windows for netcdf4==1.6.0, while 1.5.8 is fine
To report a non-security related issue, please provide:
- the version of the software with which you are encountering an issue
- environmental information (i.e. Operating System, compiler info, java version, python version, etc.)
- a description of the issue with the steps needed to reproduce it
Installing a 1.6.0 version on Windows results in a "ImportError: DLL load failed: The specified module could not be found." message. Downgrading to 1.5.8 without changing anything else solves the issue. Maybe there is a packaging problem for 1.6.0?
See below for the steps to reproduce:
➜ pip show netcdf4
Name: netCDF4
Version: 1.6.0
Summary: Provides an object-oriented python interface to the netCDF version 4 library.
Home-page: http://github.com/Unidata/netcdf4-python
Author: Jeff Whitaker
Author-email: [email protected]
License: License :: OSI Approved :: MIT License
Location: d:\miniconda3\envs\my-package\lib\site-packages
Requires: cftime, numpy
➜ python
Python 3.7.12 | packaged by conda-forge | (default, Oct 26 2021, 05:35:01) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import netcdf4
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'netcdf4'
>>> import netCDF4
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "D:\miniconda3\envs\my-package\lib\site-packages\netCDF4\__init__.py", line 3, in <module>
from ._netCDF4 import *
ImportError: DLL load failed: The specified module could not be found.
>>> exit()
➜ pip install "netcdf4~=1.5.0"
Looking in indexes: https://pypi.org/simple
Collecting netcdf4~=1.5.0
Using cached netCDF4-1.5.8-cp37-cp37m-win_amd64.whl (3.0 MB)
Requirement already satisfied: numpy>=1.9 in d:\miniconda3\envs\my-package\lib\site-packages (from netcdf4~=1.5.0) (1.21.6)
Requirement already satisfied: cftime in d:\miniconda3\envs\my-package\lib\site-packages (from netcdf4~=1.5.0) (1.6.1)
Installing collected packages: netcdf4
Attempting uninstall: netcdf4
Found existing installation: netCDF4 1.6.0
Uninstalling netCDF4-1.6.0:
Successfully uninstalled netCDF4-1.6.0
Successfully installed netcdf4-1.5.8
➜ python
Python 3.7.12 | packaged by conda-forge | (default, Oct 26 2021, 05:35:01) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import netCDF4
>>> exit()
My config: Windows 10, python 3.7.12, a conda environment, all installation performed by pip.
If you have a general question about the software, please view our Suggested Support Process.
Don't have access to a windows machine, I will try to find one next week. Note that the Windows wheels are contributed by @cgohlke (https://www.lfd.uci.edu/~gohlke/pythonlibs/).
@jswhit is there any prospect of using cibuildwheel for Windows builds? There was talk of @cgohlke's lab shutting down before July 2022. While the link is still active (and mention of the shutdown is gone), it says "Archived" and I don't think it's been updated in a couple months--for instance, it doesn't have any builds for SciPy 1.9.
How was netCDF4-1.6.0 installed for Python 3.7 on Windows? There are no netCDF4-1.6.0-cp37-*-win wheels on PyPI.
How was netCDF4-1.6.0 installed for Python 3.7 on Windows? There are no
netCDF4-1.6.0-cp37-*-winwheels on PyPI.
It is not really my domain, but here are the observations. From what I see, if I use pip, it tries to install a 1.6.0 version somehow, but there are no wheels indeed and building it fails:
➜ pip install netcdf4 --no-cache-dir
Looking in indexes: https://pypi.org/simple
Collecting netcdf4
Downloading netCDF4-1.6.0.tar.gz (774 kB)
---------------------------------------- 774.2/774.2 kB 5.4 MB/s eta 0:00:00
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [29 lines of output]
Traceback (most recent call last):
File "<string>", line 36, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "C:\Users\aserov\AppData\Local\Temp\pip-install-9gcdn49k\netcdf4_4eb5fbfbc5354ed7be80afb2fb8cd41a\setup.py", line 444, in <module>
_populate_hdf5_info(dirstosearch, inc_dirs, libs, lib_dirs)
File "C:\Users\aserov\AppData\Local\Temp\pip-install-9gcdn49k\netcdf4_4eb5fbfbc5354ed7be80afb2fb8cd41a\setup.py", line 385, in _populate_hdf5_info
raise ValueError('did not find HDF5 headers')
ValueError: did not find HDF5 headers
reading from setup.cfg...
HDF5_DIR environment variable not set, checking some standard locations ..
However, conda has a wheel for win64 and netcdf4:
➜ conda install netcdf4
Collecting package metadata (current_repodata.json): done
Solving environment: done
## Package Plan ##
environment location: D:\miniconda3\envs\test-nc
added / updated specs:
- netcdf4
The following NEW packages will be INSTALLED:
bzip2 conda-forge/win-64::bzip2-1.0.8-h8ffe710_4
cftime conda-forge/win-64::cftime-1.6.1-py37h3a130e4_0
curl conda-forge/win-64::curl-7.83.1-h789b8ee_0
hdf4 conda-forge/win-64::hdf4-4.2.15-h0e5069d_4
hdf5 conda-forge/win-64::hdf5-1.12.2-nompi_h57737ce_100
intel-openmp conda-forge/win-64::intel-openmp-2022.1.0-h57928b3_3787
jpeg conda-forge/win-64::jpeg-9e-h8ffe710_2
krb5 conda-forge/win-64::krb5-1.19.3-hc8ab02b_0
libblas conda-forge/win-64::libblas-3.9.0-16_win64_mkl
libcblas conda-forge/win-64::libcblas-3.9.0-16_win64_mkl
libcurl conda-forge/win-64::libcurl-7.83.1-h789b8ee_0
liblapack conda-forge/win-64::liblapack-3.9.0-16_win64_mkl
libnetcdf conda-forge/win-64::libnetcdf-4.8.1-nompi_h85765be_104
libssh2 conda-forge/win-64::libssh2-1.10.0-h9a1e1f7_3
libzip conda-forge/win-64::libzip-1.9.2-h519de47_1
libzlib conda-forge/win-64::libzlib-1.2.12-h8ffe710_2
mkl conda-forge/win-64::mkl-2022.1.0-h6a75c08_874
netcdf4 conda-forge/win-64::netcdf4-1.6.0-nompi_py37h45d5242_101
numpy conda-forge/win-64::numpy-1.21.6-py37h2830a78_0
tbb conda-forge/win-64::tbb-2021.5.0-h2d74725_1
Proceed ([y]/n)?
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Retrieving notices: ...working... done
I see this wheel when I uninstall and try to install the wheel with pip (I normally do not mix conda and pip installations, this is for demonstration purposes only):
➜ conda uninstall netcdf4
<...>
➜ pip install netcdf4
Defaulting to user installation because normal site-packages is not writeable
Looking in indexes: https://pypi.org/simple
Collecting netcdf4
Using cached netCDF4-1.6.0-cp39-cp39-win_amd64.whl (3.0 MB)
Collecting numpy>=1.9
Using cached numpy-1.23.2-cp39-cp39-win_amd64.whl (14.7 MB)
Collecting cftime
Using cached cftime-1.6.1-cp39-none-win_amd64.whl (163 kB)
Installing collected packages: numpy, cftime, netcdf4
WARNING: The script f2py.exe is installed in 'C:\Users\aserov\AppData\Roaming\Python\Python39\Scripts' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The scripts nc3tonc4.exe, nc4tonc3.exe and ncinfo.exe are installed in 'C:\Users\aserov\AppData\Roaming\Python\Python39\Scripts' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed cftime-1.6.1 netcdf4-1.6.0 numpy-1.23.2
So it seems like for some reason win64 wheels exist at least on conda. And pip alone also tries to install netcdf4==1.6.0, but the build fails. I am not sure why would it try to access this version if you say there are no wheels? (and strangely, we see that it installs a py39 win version, which does not make much sense to me) I would say, by default windows users would just want to install the 1.5.8 py37 version in this situation. Let me know what you think.
Sad news about the Laboratory for Fluorescence Dynamics. Christoph has made tremendous contributions to this community. To answer @dopplershift 's question, I don't have the time to invest in figuring out how to cibuildwheel to build windows wheels. It already takes way too much of my time to build macosx and linux wheels for each release. I would love for someone to step up and volunteer to take this on.
and to the OP's original issue - it seems that we understand what is going on. No binary wheels for 3.7, so pip tries to build from source and fails, I guess leaving some sort of half installed package that fails on import. Short term options are to use version 1.5.8 (the last one with python 3.7 windows wheels), or upgrade to python 3.8.
I would love for someone to step up and volunteer to take this on.
I can't promise I'll do it but I may have some time to allocate to try that next week.
and to the OP's original issue - it seems that we understand what is going on. No binary wheels for 3.7, so pip tries to build from source and fails, I guess leaving some sort of half installed package that fails on import. Short term options are to use version 1.5.8 (the last one with python 3.7 windows wheels), or upgrade to python 3.8.
Yes, it's part of the explanation, but clearly we also see that there exist a py37-win64 wheel on conda for 1.6.0, which just have gotten there somehow. Does anyone have an idea how?
I would love for someone to step up and volunteer to take this on.
I can't promise I'll do it but I may have some time to allocate to try that next week.
What should be done exactly? Figure out how to build Windows wheels and upload them?
There is a separate repo for building wheels. It currently uses multibuild to build manylinux and macos wheels. The process is somewhat complicated because there are so many C lib dependencies that have to be included in the wheels. Ideally all of this would be switched to using cibuildwheel, starting with windows first.
With the help of the awesome @isuruf we put a repo together. It is very experimental and we'll be modifying it soon. If someone could test the wheels artifacts here it would help a lot: https://github.com/ocefpaf/netcdf4-win-wheels/actions/runs/2974738134
thanks @ocefpaf and @isuruf! I don't have the ability to test these but it looks great.
thanks @ocefpaf and @isuruf! I don't have the ability to test these but it looks great.
I have downloaded a py37 wheel and tried to install it directly in a new Windows environment and it seemed to have installed fine. However, to really test it as part of a conda-forge distribution, I think I need a pre-release tag in the conda-forge channel for the 1.6.0 version. This way I could test how conda installs numpy and hdf5 together with netcdf4. With the wheels, I cannot do that (or at least I don't know how).
However, to really test it as part of a conda-forge distribution, I think I need a pre-release tag in the conda-forge channel for the 1.6.0 version. This way I could test how conda installs numpy and hdf5 together with netcdf4. With the wheels, I cannot do that (or at least I don't know how).
You should not need that, a pip install wheel-name should do the trick. I'll try that later today on a Windows laptop.
However, to really test it as part of a conda-forge distribution, I think I need a pre-release tag in the conda-forge channel for the 1.6.0 version. This way I could test how conda installs numpy and hdf5 together with netcdf4. With the wheels, I cannot do that (or at least I don't know how).
You should not need that, a
pip install wheel-nameshould do the trick. I'll try that later today on a Windows laptop.
As I said, this is what I did, but pip does not know about conda, so the numpy wheel is taken from the pypi pip repository and hdf5 was not installed altogether. So from what I understand, as long as you put this wheel on pypi, it should be fine (but with no hdf5), but it is unclear whether conda install netcdf4==1.6.0 will install the right stuff. Let me know if there's anything else I can do to test it.
Note how hdf5 was not installed in the log below. Is it normal?
➜ Downloads pip install .\netCDF4-1.6.0-cp37-cp37m-win_amd64.whl
Looking in indexes: https://pypi.org/simple
Processing c:\users\user\downloads\netcdf4-1.6.0-cp37-cp37m-win_amd64.whl
Collecting numpy>=1.9
Using cached numpy-1.21.6-cp37-cp37m-win_amd64.whl (14.0 MB)
Collecting cftime
Using cached cftime-1.6.1-cp37-none-win_amd64.whl (155 kB)
Installing collected packages: numpy, cftime, netCDF4
Successfully installed cftime-1.6.1 netCDF4-1.6.0 numpy-1.21.6
➜ Downloads python
Python 3.7.12 | packaged by conda-forge | (default, Oct 26 2021, 05:35:01) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import netCDF4
>>> exit()
As I said, this is what I did, but pip does not know about conda, so the numpy wheel is taken from the pypi pip repository and hdf5 was not installed altogether. So from what I understand, as long as you put this wheel on pypi, it should be fine (but with no hdf5), but it is unclear whether
conda install netcdf4==1.6.0will install the right stuff. Let me know if there's anything else I can do to test it.
@Alexander-Serov the wheel is not a conda package. It bundles all those dependencies with it and pulls the Python dependencies from PyPI.
Things worked as expected so far. Can you try to load some data with that?
@jswhit I tested these wheels against the official tests and this notebook to check if xarray and others would be OK with it. Here is how I created the test env:
conda create --name TEST "python=3.10" gh pip
conda activate TEST
pip install netCDF4-1.6.0-cp310-cp310-win_amd64.whl matplotlib xarray erddapy jupyter nc_time_axis pytest cython
gh repo clone Unidata/netcdf4-python
cd netcdf4-python\test
del tst_alignment.py # unreleased feature
python run_all.py
Here are the result from the tests:
not running tst_compression_quant.py ...
not running tst_compression_zstd.py ...
not running tst_compression_bzip2.py ...
not running tst_compression_blosc.py ...
not running tst_compression_szip.py ...
netcdf4-python version: 1.6.0
HDF5 lib version: 1.12.1
netcdf lib version: 4.8.1
numpy version 1.23.2
cython version 0.29.32
.EE.EE...........................................................................................
======================================================================
ERROR: runTest (tst_atts.VariablesTestCase)
testing attributes
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\conda-forge\netcdf4-python\netcdf4-python-master\test\tst_atts.py", line 183, in runTest
ncdump_output = f.tocdl()
File "src\netCDF4\_netCDF4.pyx", line 3355, in netCDF4._netCDF4.Dataset.tocdl
File "C:\Users\filipe\miniconda3\envs\TEST\lib\subprocess.py", line 501, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\Users\filipe\miniconda3\envs\TEST\lib\subprocess.py", line 969, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Users\filipe\miniconda3\envs\TEST\lib\subprocess.py", line 1438, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified
======================================================================
ERROR: runTest (tst_atts.VariablesTestCase)
testing attributes
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\conda-forge\netcdf4-python\netcdf4-python-master\test\tst_atts.py", line 137, in tearDown
os.remove(self.file)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'tst_atts.nc'
======================================================================
ERROR: test_fromcdl (tst_cdl.Test_CDL)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\conda-forge\netcdf4-python\netcdf4-python-master\test\tst_cdl.py", line 41, in setUp
f.tocdl(outfile='ubyte.cdl',data=True)
File "src\netCDF4\_netCDF4.pyx", line 3355, in netCDF4._netCDF4.Dataset.tocdl
File "C:\Users\filipe\miniconda3\envs\TEST\lib\subprocess.py", line 501, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\Users\filipe\miniconda3\envs\TEST\lib\subprocess.py", line 969, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Users\filipe\miniconda3\envs\TEST\lib\subprocess.py", line 1438, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified
======================================================================
ERROR: test_tocdl (tst_cdl.Test_CDL)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\conda-forge\netcdf4-python\netcdf4-python-master\test\tst_cdl.py", line 41, in setUp
f.tocdl(outfile='ubyte.cdl',data=True)
File "src\netCDF4\_netCDF4.pyx", line 3355, in netCDF4._netCDF4.Dataset.tocdl
File "C:\Users\filipe\miniconda3\envs\TEST\lib\subprocess.py", line 501, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\Users\filipe\miniconda3\envs\TEST\lib\subprocess.py", line 969, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Users\filipe\miniconda3\envs\TEST\lib\subprocess.py", line 1438, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified
----------------------------------------------------------------------
Ran 96 tests in 33.411s
FAILED (errors=4)
I'm pretty sure those failures are not related to the wheel but with Windows file permissions. Note that conda was only used to install python 3.10, everything else was installed with pip.
Note that conda was only used to install python 3.10, everything else was installed with pip.
It would be better to use a windows installer for this test to make sure the conda python dependencies are not used.
It would be better to use a windows installer for this test to make sure the conda python dependencies are not used.
Indeed. I also want to run those these in the repo we are creating the wheel. The delay on the former b/c my Windows partition decided to no longer boot up after a Windows upgrade :-/
I'm getting blue screen and re-boot cycle now :-(