Missing dependencies in pyproject.toml

[surgura]

pyproject.toml is missing dependencies. These are present in environment.yml. See the following list:

cgen
dask
cftime
psutil
netCDF4
zarr
tqdm
pymbolic (not sure but I think this is actually a missing cgen dependency?)

As an example, adding them can be done like so:

[project]
(...)
dependencies = [
"cgen",
"dask >= 2.0",
]

[erikvansebille]

Thanks for reporting @surgura. As I also mentioned in #1553, I don't have enough experience with the pypi workflow to extensively test or maintain it.

In particular, I wouldn't know how to create a workflow (Github Action?) that checks for every new release whether a pip install parcels is successful, and whether parcels indeed works. Something similar to the code in the conda-forge recipe:

- curl https://raw.githubusercontent.com/OceanParcels/parcels/master/docs/examples/example_peninsula.py > example_peninsula.py
- python example_peninsula.py

Help is very welcome!

[andrew-s28]

The main issue I see with the PyPi release is that OceanParcels depends on a conda-forge only package, specifically c-compiler (for Unix) and m2w64 (for Windows). These are not going to be possible to include as a dependency on a PyPi package (see, e.g., this discussion). I'd be happy to write a PR that updates the PyPi install with all the other packages that can be included, though.

That said, there may be some workaround I'm unaware of since I'm not really familiar with the dynamic C code generation within this project. If @erikvansebille or other folks think it's important to support a PyPi release, then perhaps there's a way to include the compilers as an intrinsic part of the package, or even just prompt the user to install the relevant C compiler if it is not found at package install time, rather than just throwing an error at runtime.

On the other hand, the only place I can find that pip install parcels is recommended to be used is in the development workflow, and that workflow already explicitly includes all the required dependencies through the creation of a conda environment and the use of environment.yaml, so it may just be fine as is.

[surgura]

Sorry for the slow response. I also do not see any PyPI package that provides a C compiler. I think it is reasonable to expect a user of pyproject.toml to install their own C compiler (For sure there are other package that have this issue is as well). It would be great if for now all packages except for the C compiler in the pyproject.toml would be kept up to date with the conda counterpart :) On my Linux system everything works well if I install those missing packages and a C compiler manually.

Regarding testing: The only way would be to run the tests twice. Once with the conda installation and once with pip, which could be done through a pipeline strategy setup. But you can choose to just not do it for pip and do a best effort at keeping the pyproject.toml up to date. It's already there anyway, we simply do an update of the packages now and then forget about the rest :) (of course once in a while an issue will pop up)

[andrew-s28]

No problem, thanks for the response!

I agree with your point @surgura re: dependencies, it is not much effort to include them in pyproject.toml and will make for a more complete package, even without a C compiler dependency.

Re: testing, I'm not sure exactly what you mean by running the tests twice with a pipeline strategy, but I did include a test of the PyPi release through the publishing to PyPi workflow in the PR linked above. Let me know if you think that this is a good method for testing the release or not.

[surgura]

I meant running the units tests a second time, but instead of using conda you use pip. A downside of python is that you never know if your installation is correct until you run the whole thing :)

[erikvansebille]

I just released the v3.0.3 update with the updated pyproject.toml and Action workflow, and this appears to have worked! See https://github.com/OceanParcels/parcels/actions/runs/9264397852

[surgura]

nice!