BPCells icon indicating copy to clipboard operation
BPCells copied to clipboard

Python PyPI deploy setup

Open bnprks opened this issue 1 year ago • 1 comments

This is a large set of changes, described below with a summary of per-file changes. The end result is that can build and deploy a very basic BPCells package to PyPI across Mac, Linux and Windows (and ensure tests pass before deploy). There are also several docs updates to help add a Python section to the documentation website

Deploy to PyPI:

  • .github/workflows/pypi.yml (lots of changes done after using a test repo to figure out much of the needed changes without waiting for slow build times)
  • python/pyproject.toml Move all the cibuildwheel config into the pypi.yml workflow, since I felt a lot of that workflow pulled in specifics about the github CI environment
  • python/scripts/install_*.sh Helper scripts for CI to install dependencies from source (or ccache from binary)
  • python/setup.py Switch to a more flexible way of getting additional compiler arguments from environment variables (just CPATH and LIBRARY_PATH and removing HWY_* flags). Also fix up the logic for restoring messed up symlinks on windows checkouts, to include the new vendor folder

Updated docs:

  • DEVELOPING.md Update instructions for generating docs website
  • README.md Make parallel changes to r/index.md, though slight changes given that this is the github landing page
  • python/DEVELOP.md (Takes most of the old README content and adds a section on the CI setup)
  • python/README.md (Trying to be a short README as the PyPI landing page)
  • python/docs/source/index.md , python/docs/source/python.md Try to mostly unify these two pages -- they are only separate due to the PyData Sphinx theme setting the top nav bar based on the table of contents of the landing page
  • r/index.md Hide the detailed compilation instructions behind a <details> dropdown, and add a call-out to the Python docs site up top
  • r/pkgdown/_pkgdown.yml, r/pkgdown/extra.css. Add a link to the Python docs in the website header, with a slight CSS styling adjustment to make it not too big

Misc changes:

  • python/src/fragments.cpp: One of the tests failed on Windows, due to temporary files still being open at the time of deletion in a C++ function. (Windows is a bit unique in that it disallows deleting open files). Just added in a line to cause the appropriate C++ destructor to be called to close temp files prior to deleting them
  • r/src/bpcells-cpp/arrayIO/binaryfile.h: Add a quick compatibility check for the Windows MSVC compiler to call the correct name for the builtin 32-bit byteswap function.

bnprks avatar Aug 26 '24 20:08 bnprks

I consolidated to a single linux/mac script to build dependencies from source, then switched to recommending that in DEVELOP.md for a simpler (albeit slower) setup process.

Also made a few changes in the actions workflow to change how caching the dependencies would work, and after much trial and error it seems to be functional again.

bnprks avatar Aug 30 '24 05:08 bnprks