Build wheels for Python >=3.8 on Linux

[ecederstrand]

pip install pytraj fails on Python 3.8 with Detected use of pip; you must set CPPTRAJHOME if there is no cpptraj in the current folder.

This is because there are no wheel files for cp38 at https://pypi.org/project/pytraj/#files so pip tries to install from source. Can you please upload wheel files for Python 3.8, or release a new version that supports Python 3.8?

On a side note, setup.py checks for Python 2.6 or later, but the classifiers section states Python 2.7 (and Python 3.7 and 3.8 are missing): https://github.com/Amber-MD/pytraj/blob/75070a9ab2926b89c9cf1367ff9d3a648702b68f/setup.py#L239

[ecederstrand]

With #1516, this is just waiting for a new release of pytraj to PyPI :-)

[ecederstrand]

Hi @hainm Would you be able to cut a new release of pytraj and upload new packages to PyPI?

[hainm]

hi @ecederstrand, I am waiting for AmberTools to be released (soon) to get the final version of cpptraj.

[ecederstrand]

Cool. Thanks!

I apologize for my eagerness. I'm maintaining a JupyterHub installation with almost 700 Python packages, and this is the last one missing Python 3.8 support :-)

[ecederstrand]

@hainm What is the current procedure to build a pip wheel from pytraj source? I'm reading https://github.com/Amber-MD/pytraj/blob/master/DEVELOPMENT.md but build_pip.sh has been moved to the "deprecated" folder: https://github.com/hainm/pytraj-build-box/tree/master/deprecated so I'm wondering if the instructions are outdated.

I tried to run build_pip.sh anyway but it calls https://github.com/Amber-MD/pytraj/blob/master/scripts/build_wheel.py which seems to also require a conda environment. I don't have conda installed. Do I need it even though I'm not targeting conda?

Looking at the Docker setup in https://github.com/hainm/pytraj-build-box, it looks like running the Docker image will build packages for Python 3.5 which is not what I want.

[hainm]

@ecederstrand For release, I will run this script on my mac: https://github.com/Amber-MD/pytraj/blob/master/devtools/builder/build_all.sh

The script will build releases for both pip and conda and for both linux and macos.

I tried to run build_pip.sh anyway but it calls https://github.com/Amber-MD/pytraj/blob/master/scripts/build_wheel.py which seems to also require a conda environment. I don't have conda installed. Do I need it even though I'm not targeting conda?

Yes. conda is used to create different python versions.

Looking at the Docker setup in https://github.com/hainm/pytraj-build-box, it looks like running the Docker image will build packages for Python 3.5 which is not what I want.

The build script uses the python version in /opt/python/<python tag> folder. But that image does not have python 3.8 yet.

So here is the process I would do:

• Update the docker image to use the latest image from manylinux that has python 3.8 https://github.com/pypa/manylinux
• Update python 3.8 in here: https://github.com/Amber-MD/pytraj/blob/1fea8ffd2163fd93f88dd5f3c0b4072aee51c63a/devtools/builder/build_all.sh#L47 and here: https://github.com/Amber-MD/pytraj/blob/1fea8ffd2163fd93f88dd5f3c0b4072aee51c63a/scripts/build_wheel.py#L35
• Run the build_all.sh script or https://github.com/Amber-MD/pytraj/blob/master/devtools/builder/run_docker_build_wheels_linux.sh

You can tell that the process is quite complicated and it's not really smooth (used to be smooth). That's why I don't have time to make 3.8 release yet. In the future I might move the build process to conda-forge, but it takes time too. (My defense: I am quite busy recently with my daily work and my kid :D).

[ecederstrand]

Thanks for the pointers! I'll give it a go and let you know.

I very much understand that daily work and family take priority over open source work :-)

[hainm]

Thanks for the pointers! I'll give it a go and let you know.

Cool, thanks. Let me know if you have any further questions.

[ecederstrand]

I can't see traces of the 2.0.5 release here on GitHub, but I have confirmed that the Python 3.8 packages for 2.0.5 on PyPI work for us. Thanks for all your hard work!

[ecederstrand]

Hmm, I must have messed up my test somehow. There are only OS X packages for 3.8 on PyPI.

[ecederstrand]

With the slight change in https://github.com/ecederstrand/pytraj/commit/881b4478ae9048813a0790391279350c505d518f, I was able to build binary packages for Python 3.8 without the Docker image, using the standard build_ext bdist_wheel command:

git clone https://github.com/ecederstrand/pytraj.git
cd pytraj
git checkout 881b4478ae9048813a0790391279350c505d518f
git clone https://github.com/Amber-MD/cpptraj.git
python3.8 -m pip install wheel Cython
python3.8 setup.py build_ext bdist_wheel

[hainm]

Hmm, I must have messed up my test somehow. There are only OS X packages for 3.8 on PyPI.

yeah, there are only wheels for python 3.5, 3.6, 3.7 for linux for now.

With the slight change in ecederstrand@881b447, I was able to build binary packages for Python 3.8 without the Docker image, using the standard build_ext bdist_wheel command:

There was no issue with building original wheel for pytraj in any system (it means you don't need docker image at all).

The only issue here is that pytraj needs auditwheel to repair the original wheel so the resulted wheel can be used in most of the linux distro. And that needs to be done in an old linux distro (pytraj has been using centos-5) so pytraj can be used in a newer distro. That's why pytraj used the docker image.

Since you're using the original wheel, make sure to add cpptraj/lib/libcpptraj.so in your LD_LIBRARY_PATH (this also means if you want to ship pytraj, you need to ship libcpptraj.so too. The repaired wheel via auditwheel would handle that. :D).

export LD_LIBRARY_PATH=`pwd`/cpptraj/lib:$LD_LIBRARY_PATH

[markperri]

Is there a package for python3.8 available? I'd like to upgrade Jupyter Notebook, but they're on 3.8 now.

[hainm]

hi @markperri: unfortunately there is no wheel 3.8 for Linux (macs does).

However, you can install ambertools from conda-forge that includes pytraj.

conda install -c conda-forge ambertools=20

[markperri]

@hainm Thanks! I didn't realize there was a conda package. That will make life simpler. Thanks, Mark

[Seanny123]

You could try using cibuildwheels to build the wheels automatically for all supported Python versions for each release. This is what MDAnalysis is trying to do, so maybe talk to them if you want to adopt it for this project.

[hainm]

Thank you very much.

Hai

On Fri, May 20, 2022 at 12:54 PM Sean Aubin @.***> wrote:

You could try using cibuildwheels https://github.com/pypa/cibuildwheel to build the wheels automatically for all supported Python versions for each release. This is what MDAnalysis is trying to do https://github.com/MDAnalysis/mdanalysis/issues/1300#issuecomment-1077902172, so maybe talk to them if you want to adopt it for this project.

— Reply to this email directly, view it on GitHub https://github.com/Amber-MD/pytraj/issues/1508#issuecomment-1133120842, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABB645NZ4CA4YOJZCJRS3UTVK67UNANCNFSM4KKWBJ4A . You are receiving this because you were mentioned.Message ID: @.***>