opensim-core icon indicating copy to clipboard operation
opensim-core copied to clipboard

Add BUILD_PYTHON_WHEELS option, use it to control installation of libraries int…

Open aymanhab opened this issue 3 months ago • 15 comments

…o sdk/Python folder, also modify setup.py Fixes issue #<issue_number>

Brief summary of changes

Modify build system to support building python wheels for pypi distribution. Involved some minor cmake refactoring, but ended up making it as a "post install" step on mac, linux to minimize changes to flaky rpath monsters. Wheels on windows do build and install but fail to load one casadi dll. Disabling in ci

Testing I've completed

Published wheels and tested, had others test installation locally, beefed up ci to test after installing wheels.

Looking for feedback on...

CHANGELOG.md (choose one)

  • updated.

This change is Reviewable

aymanhab avatar Nov 07 '25 23:11 aymanhab

@davidpagnon The latest artifact has wheels to install latest opensim on all platform with python 3.10, can you give this a spin, as we'd love to hear your thoughts? If you have a specific python version I can easily modify the build script to accommodate and create a wheel as we need more testers. Thank you

aymanhab avatar Nov 18 '25 18:11 aymanhab

@nickbianco @moorepants just a heads up that the wheels are available on the artifact.

aymanhab avatar Nov 18 '25 18:11 aymanhab

@nicos1993 @stingjp

aymanhab avatar Nov 18 '25 19:11 aymanhab

This is a good approach, given that you can bundle all dependencies into the wheel.

Here is my attempt at installing the linux whl:

$ conda create -n opensim python=3.10 numpy pip setuptools
Channels:
 - conda-forge
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/moorepants/miniconda/envs/opensim

  added / updated specs:
    - numpy
    - pip
    - python=3.10
    - setuptools


The following NEW packages will be INSTALLED:

  _libgcc_mutex      conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge 
  _openmp_mutex      conda-forge/linux-64::_openmp_mutex-4.5-2_gnu 
  bzip2              conda-forge/linux-64::bzip2-1.0.8-hda65f42_8 
  ca-certificates    conda-forge/noarch::ca-certificates-2025.11.12-hbd8a1cb_0 
  icu                conda-forge/linux-64::icu-75.1-he02047a_0 
  ld_impl_linux-64   conda-forge/linux-64::ld_impl_linux-64-2.45-h1aa0949_0 
  libblas            conda-forge/linux-64::libblas-3.11.0-1_h4a7cf45_openblas 
  libcblas           conda-forge/linux-64::libcblas-3.11.0-1_h0358290_openblas 
  libexpat           conda-forge/linux-64::libexpat-2.7.1-hecca717_0 
  libffi             conda-forge/linux-64::libffi-3.5.2-h9ec8514_0 
  libgcc             conda-forge/linux-64::libgcc-15.2.0-h767d61c_7 
  libgcc-ng          conda-forge/linux-64::libgcc-ng-15.2.0-h69a702a_7 
  libgfortran        conda-forge/linux-64::libgfortran-15.2.0-h69a702a_7 
  libgfortran5       conda-forge/linux-64::libgfortran5-15.2.0-hcd61629_7 
  libgomp            conda-forge/linux-64::libgomp-15.2.0-h767d61c_7 
  liblapack          conda-forge/linux-64::liblapack-3.11.0-1_h47877c9_openblas 
  liblzma            conda-forge/linux-64::liblzma-5.8.1-hb9d3cd8_2 
  libnsl             conda-forge/linux-64::libnsl-2.0.1-hb9d3cd8_1 
  libopenblas        conda-forge/linux-64::libopenblas-0.3.30-pthreads_h94d23a6_4 
  libsqlite          conda-forge/linux-64::libsqlite-3.51.0-hee844dc_0 
  libstdcxx          conda-forge/linux-64::libstdcxx-15.2.0-h8f9b012_7 
  libstdcxx-ng       conda-forge/linux-64::libstdcxx-ng-15.2.0-h4852527_7 
  libuuid            conda-forge/linux-64::libuuid-2.41.2-he9a06e4_0 
  libxcrypt          conda-forge/linux-64::libxcrypt-4.4.36-hd590300_1 
  libzlib            conda-forge/linux-64::libzlib-1.3.1-hb9d3cd8_2 
  ncurses            conda-forge/linux-64::ncurses-6.5-h2d0b736_3 
  numpy              conda-forge/linux-64::numpy-2.2.6-py310hefbff90_0 
  openssl            conda-forge/linux-64::openssl-3.6.0-h26f9b46_0 
  pip                conda-forge/noarch::pip-25.3-pyh8b19718_0 
  python             conda-forge/linux-64::python-3.10.19-h3c07f61_2_cpython 
  python_abi         conda-forge/noarch::python_abi-3.10-8_cp310 
  readline           conda-forge/linux-64::readline-8.2-h8c095d6_2 
  setuptools         conda-forge/noarch::setuptools-80.9.0-pyhff2d567_0 
  tk                 conda-forge/linux-64::tk-8.6.13-noxft_ha0e22de_103 
  tzdata             conda-forge/noarch::tzdata-2025b-h78e105d_0 
  wheel              conda-forge/noarch::wheel-0.45.1-pyhd8ed1ab_1 
  zstd               conda-forge/linux-64::zstd-1.5.7-hb8e6e7a_2 



Downloading and Extracting Packages:

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate opensim
#
# To deactivate an active environment, use
#
#     $ conda deactivate

$ conda activate opensim 
(opensim) $ python -m pip install -v ~/Downloads/opensim-4.5.2-py3-none-any.whl 
Using pip 25.3 from /home/moorepants/miniconda/envs/opensim/lib/python3.10/site-packages/pip (python 3.10)
Processing ./Downloads/opensim-4.5.2-py3-none-any.whl
Requirement already satisfied: numpy<=2.3,>=2.1 in ./miniconda/envs/opensim/lib/python3.10/site-packages (from opensim==4.5.2) (2.2.6)
Installing collected packages: opensim
  changing mode of /home/moorepants/miniconda/envs/opensim/bin/opensim-moco-generate-report to 775
Successfully installed opensim-4.5.2
(opensim) $ python
Python 3.10.19 | packaged by conda-forge | (main, Oct 22 2025, 22:29:10) [GCC 14.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import opensim
>>> v = opensim.Vector([6, 7, 8, 9])
>>> v
<opensim.simbody.Vector; proxy of <Swig Object of type 'SimTK::Vector_< double > *' at 0x796c89ee5d40> >

Seems to work.

moorepants avatar Nov 18 '25 19:11 moorepants

I get the same result as @moorepants on Mac Arm64.

nickbianco avatar Nov 18 '25 20:11 nickbianco

It seems you have the wheels with the platform tag as any, these will need to follow the correct format for uploading to PyPi: https://peps.python.org/pep-0491/#file-name-convention

moorepants avatar Nov 18 '25 20:11 moorepants

Tested the Windows wheels, works.

nicos1993 avatar Nov 18 '25 22:11 nicos1993

Worked following the same commands on my machine as well (Mac Arm64).

stingjp avatar Nov 18 '25 22:11 stingjp

  • Naive question, is there a reason why the files are called pyproject.tom1 and setup.py.in instead of pyproject.toml and setup.py? Cloning the repository and running pip install . did not work, even after changing the names.

  • I did a wild test on my (empty) python 3.12 environment after downloading the wheel from there: https://github.com/opensim-org/opensim-core/actions/runs/19479727468 The installation worked, but then importing opensim resulted in this error:

    import opensim as osim
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "C:\Users\david\miniconda3\envs\openSim\Lib\site-packages\opensim\__init__.py", line 19, in <module>
        from .simbody import *
      File "C:\Users\david\miniconda3\envs\openSim\Lib\site-packages\opensim\simbody.py", line 10, in <module>
        from . import _simbody
    ImportError: cannot import name '_simbody' from partially initialized module 'opensim' (most likely due to a circular import) (C:\Users\david\miniconda3\envs\openSim\Lib\site-packages\opensim\__init__.py)
    
  • Same outcome with a Python 3.10 environment.

I might be doing something wrong, but I don't have time to further investigate right now In any case, this is exciting! It will make installation much easier in certain cases, on Blender for example! (pip install instead of replacing the whole Blender python by a hacky conda environment)

davidpagnon avatar Nov 19 '25 08:11 davidpagnon

It seems you have the wheels with the platform tag as any, these will need to follow the correct format for uploading to PyPi: https://peps.python.org/pep-0491/#file-name-convention

This has been fixed in the latest iteration, thanks for the feedback @moorepants Any further feedback is much appreciated.

aymanhab avatar Nov 21 '25 17:11 aymanhab

@nickbianco Ready for review once ci build finishes.

aymanhab avatar Nov 21 '25 17:11 aymanhab

Do you know if it's possible to upload multiple wheels with one action? I guess we would need to revamp the CI if we wanted to build wheels for multiple python versions anyway.

This is a common tool used to build the correct wheels across python versions and operating systems: cibuildwheel.pypa.io/en/stable/

moorepants avatar Nov 23 '25 05:11 moorepants

@nickbianco Let me clarify, the deleted files were never used as the only way for the tests to pass without copying all the data files and models to the package folder is to run them from the source tree as we have done in the past, we do this (run tests from different folder) on main branch and we're doing it now in this PR. We probably should remove the copying of this opensim/tests folder to the package folder to avoid misleading future developers but functionality/testing-wise we lost absolutely nothing as you can see by checking the ci build log on main. With that if you prefer I restore these unused files I'm ok with that too.

aymanhab avatar Nov 26 '25 17:11 aymanhab

@aymanhab, if we do decide to remove the tests, I think we should at the very least handle that on a separate PR. I understand that they might not be currently used, but if there is value in reinstating the tests, then I would like to consider it. Is is possible to revert the test changes on this branch without holding up the PR?

nickbianco avatar Nov 26 '25 21:11 nickbianco

Test files restored. Thanks for the review @nickbianco 👍

aymanhab avatar Nov 27 '25 01:11 aymanhab

Squashed and merged, thanks for the review and great feedback @nickbianco @moorepants

aymanhab avatar Dec 16 '25 19:12 aymanhab

Awesome, does it mean that we are going to have OpenSim on Pypi pretty soon?

Also, I just came across pyopensim, do you know about it? https://pypi.org/project/pyopensim/

davidpagnon avatar Dec 16 '25 22:12 davidpagnon

@davidpagnon yes we plan to have opensim on pypi after working through some issues with package ownership etc. pyopensim was the motivation to revisit pypi, they do a great job but it's much easier for us to build the wheels while building opensim. Also pyopensim doesn't contain moco while our wheels do, so stay tuned. If you're on mac or linux you maybe able to use the wheels now from build artifacts. Still working on the windows side.

aymanhab avatar Dec 16 '25 22:12 aymanhab