DIST/DEV: Improve method of building packages in development

[asoplata]

For our eventually-automated package building, George recommended that we use a modern python package manager like poetry, uv, or pdm. Note that is has NO impact on the user’s install, and instead is only for making things more convenient for us developers. See details below.

Note that this is related to, but not the same as, #950 . That said, we can make the changes within this current Issue without requiring the changes in #950 .

Subtasks

• [ ] Investigate uv, pdm, poetry, and others
• [ ] Test building a simple package with whatever we choose
• [ ] Incorporate the builder package into Github Actions that automate the release process (George’s meeting notes below are especially helpful for this)
• [ ] Setup “Trusted Publishers” for better security (no personal Pypi API tokens, etc.)

Notes

• George specifically provided very useful information to Dylan and Austin about this in this meeting: https://www.notion.so/jonescompneurolab/HNN-Core-Improvements-Meeting-with-George-Dang-abc4a36e4c844b38ba81da39122ab35a
• Notably, poetry is popular and has been around the longest, BUT uses a nonstandard pyproject.toml (pdm's documentation points this out here https://github.com/pdm-project/pdm?tab=readme-ov-file#poetry ). Austin thinks we should avoid it for this reason alone, and is particularly in favor of uv.
• This has many benefits:
  • Improved speed when installing locally (who cares)
  • …improved speed when installing which is also relevant for our CI runners! As of 2024-11-21, each testing CI run uses, very loosely, around 20% of its runtime just installing hnn-core. This may be significantly reduced if we use something like uv .
  • These can also be used to manage your Python versions AND virtual environments, reducing the need to rely on an external tool like conda or pyenv etc. For a new developer, we may be able to get their environment setup in very few commands (2?).
  • Enables use of lockfiles and similar methods at controlling consistency between developer installs. This way, all developers can both
    1. trust that they are using the same versions of dependencies as other developers by default, but also
    2. test against an explicit version change in a dependency if there is an incompatibility, and use this same method to communicate the dependency version change itself to other devs
  • To be clear and verbose, this would also replace our use of dependencies for the building itself, such as replacing twine https://github.com/astral-sh/uv?tab=readme-ov-file#highlights
  • We can merge our independent doc building pipeline using the same tooling we use to create our actually packaged code. We would no longer need 2 different Makefiles that are managed differently.

[jasmainak]

I have used poetry in the past and did not like it :) The builds just take too long and get hung, but maybe it has improved. Again, same message ... look around what other repos in the python scientific ecosystem are doing and copy !

[asoplata]

This has been resolved. Unfortunately, as much as I want to use the new and shiny uv package, it is insufficient for HNN's purposes due to the optional OpenMPI dependency. As a result, we now recommend Anaconda use by default for HNN, including that we now provide conda packages for HNN itself which simplify MPI installation. See installation guide here https://jonescompneurolab.github.io/textbook/content/01_getting_started/installation.html . See documentation on how to build our current releases here https://jonescompneurolab.github.io/hnn-core/dev/how_to_create_releases.html .

Additionally, we have updated and activated our Github Actions so that they can build our PyPI packages inside the Actions themselves, followed by automatic uploading to the PyPI index using Trusted Publishing. See #1073, #1077, #1081, and #1090.

Due to our Anaconda dependency for MPI, especially for developers, at this time I don't think we need to make major changes for anything that Anaconda does NOT support (like lockfiles).

[jasmainak]

can you not do:

uv pip install mpi4py openmpi

See here

[asoplata]

Wow! This changes everything!!!

It appears that the developers of mpi4py have begun maintaining a pip-installable version of OpenMPI, starting 3 months ago https://pypi.org/project/openmpi/#files . I didn't catch this because it is relatively recent - I definitely have poured over the mpi4py Installation page you linked frequently and in detail over the last year, but not since I built our conda packages in May (see #950).

This is fantastic, because it means that if their pip openmpi package works, we can drop our conda packaging entirely, and forget about needing conda for our OpenMPI installation! We could exclusively use pip and uv for developer and user installation.

Even more exciting is that if we don't need conda anymore for either OpenMPI installation or packaging, then we can start shipping binary wheels! We can pre-compile the MOD files that users need across platforms (using local or Github Actions), then ship a PyPI wheel that works with the pip openmpi package. (We could probably do this already, but previously it wasn't worth the time: we were already pre-compiling platform-specific conda packages for the same purpose.) This is especially nice since it means users probably (?) won't need to download compilers like GCC (Linux) or Xcode Command-Line Tools (Mac) just to run a simulation.

Thanks for bringing my attention to this Mainak! If the pip openmpi package works on all our platforms (which it almost certainly does), then both myself and new users installing HNN will be much happier :)