crossenv icon indicating copy to clipboard operation
crossenv copied to clipboard

Upgrade to PEP 621, and add GitHub Actions configuration

Open freakboy3742 opened this issue 1 year ago • 4 comments

Stemming from #117, this PR attempts to restore CI configuration, independent of any other changes.

It upgrades the project configuration to PEP 621 format (i.e., pyproject.toml). A pyproject.toml file with a PEP517 configuration is required to avoid warnings (and, in the very near future installation errors); migrating the rest of the configuration was relatively trivial.

It adds a tox configuration to make it easier to run tests for multiple Python versions; and adds a GitHub Actions configuration to invoke that tox configuration across a matrix of Python versions.

Coverage reporting is done in the build, and a HTML report is uploaded as a build artefact.

freakboy3742 avatar Sep 10 '24 03:09 freakboy3742

@benfogle FYI - it's difficult to finalize this PR until GitHub Actions are actually running. I think this is everything that is needed, but I think actions won't actually run until there's at least a stub actions configuration in the main branch.

freakboy3742 avatar Sep 10 '24 03:09 freakboy3742

@benfogle I've triggered this CI configuration in my own repo; however, it's currently failing, with the same (or similar) errors to what I'm seeing locally.

You've mentioned that you're able to run the tests on Ubuntu 22.04 - however, I'm not sure I see how (at least, not without some extensive apt repo customization). The problems seen in the CI runs all stem from the fact that the 2 prebuilt environments (prebuilt_musl_arm_aarch64.tar.xz and prebuild_python3.9.0.tar.xz) have been built on (I think) Ubuntu 18.04. As a result, they reference libssl1.1.so and libffi6.so. These libraries are no longer available on Ubuntu 22.04 (OpenSSL 1.1.1 is unmaintained, and 22.04 ships with libffi7.so).

Ubuntu 18.04 is no longer available as a runtime environment on GitHub Actions, so that's not a good option for CI. However, it's not clear to me how the prebuilt environments were built (or even if that's something you'd want to be contributed as part of a PR, due to the potential security risk of binaries from third party).

Are you able to advise:

  1. How the prebuilt environments are built?
  2. How you'd like to proceed with updating those prebuilt environments for compatibility with Ubuntu 22.04 (or 24.04, given that is the current ubuntu-latest environment is in GitHub Actions).

freakboy3742 avatar Oct 01 '24 06:10 freakboy3742

@benfogle I've now tried running the test suite under Docker using Ubuntu 18.04, and I haven't had any more luck.

Most of these problems stem from the fact that the 18.04 ships with Python 3.6 - and more recent versions of pip and setuptools aren't backwards compatible that far. As a result, the numpy and bcrypt test_basic tests fail because the setup.py configurations of the specific packages that are referenced aren't compatible with recent versions of Python, and you end up in the general pre-PEP 517 hell of finding a combination of Python, pip, and setuptools that are mutually compatible.

I've tried rebuilding the prebuilt packages locally under 22.04, and didn't have any luck, either. My first attempt failed because the URL for zlib downloads has changed; on the second attempt, after an hour, the build failed at 90% trying to configure host-python-main-armhf, with an error complaining about a missing --with-build-python argument. From the look of it, that's because it's trying to build the main branch of Python with a set of configure arguments that haven't been sufficient since Python 3.10.

So - I'm not sure where to take things from here. It appears there's a lot of work needed to restore CI to a working state; however, I'm not sure of the best way to contribute that work. Any tips, suggestions or preferences would be gratefully accepted at this point.

freakboy3742 avatar Oct 02 '24 04:10 freakboy3742

Thanks for this. You're 100% right that more needs to change to restore CI. I need to poke around at a few options and figure out what's best. It will probably involve converting the git lfs artifacts into docker images. I will likely base that work of off this branch. Thanks for being patient with this.

benfogle avatar Oct 06 '24 00:10 benfogle

@freakboy3742 I've taken over as maintainer of this project, in the very near future I will be putting together CI for crossenv patterned after what I do for the RobotPy project... so I'll likely close this PR in favor of whatever I end up with -- but thanks for taking a crack at this first.

As an initial push, look for docker images at https://github.com/robotpy/crossenv-ci-images in the near future (maybe tonight, more likely tomorrow evening)

virtuald avatar Sep 09 '25 05:09 virtuald

@virtuald Congratulations on the new project ownership.

I'm happy to pre-emptively close this PR for you. FWIW, I've been pursuing different options in the cross-platform build space. That project draws from the lessons learned adding cross-platform support in cibuildwheel for iOS and Android (and, tangentially, from working on Emscripten); what I found was that the approach taken by crossenv was massive overkill for those use cases. I can't speak to what utility you've got for your own usage, but that's where I'll be focussing efforts for the foreseeable future.

freakboy3742 avatar Sep 09 '25 05:09 freakboy3742

I really like the concept, definitely seems like it could be a much simpler approach. Look forward to seeing it develop more.

virtuald avatar Sep 09 '25 05:09 virtuald

I've got images up at https://github.com/robotpy/crossenv-ci-images and github actions CI, you might find the images useful for xbuild (though they are overkill).

virtuald avatar Sep 12 '25 06:09 virtuald