dockstore
Investigate instructions with Python 3.12.3
Describe the bug Linux quick start instructions working, but may have issues on Mac. Take a look and see if reproducible
If it helps, the Ubuntu command-history and versions that worked were
apt-get install curl
curl -o requirements.txt "https://dockstore.org/api/metadata/runner_dependencies?client_version=1.15.2&python_version=3"
apt-get install python3
apt-get install python3-pip
pip3 install -r requirements.txt
cwltool --version
root@e41b7fe60ff9:/# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS"
root@e41b7fe60ff9:/# cwltool --version
/usr/local/bin/cwltool 3.1.20230201224320
root@e41b7fe60ff9:/# python3 --version
Python 3.10.12
root@e41b7fe60ff9:/# pip3 --version
pip 22.0.2 from /usr/lib/python3/dist-packages/pip (python 3.10)
To Reproduce Try it on Mac
Additional context [Domain] - https://dockstore.org
- 1.15.2
- 2.12.2
- 1.15.0
┆Issue is synchronized with this Jira Story ┆Fix Versions: Dockstore 1.15.x ┆Issue Number: DOCK-2539 ┆Sprint: 145 - Richmond ┆Issue Type: Story
https://pypi.org/project/cwltool/#description says it only works up to 3.11, user was running 3.12 -- may not be Mac-specific as the above Linux test was with Python 3.10.
Ah Python. Minor updates look not so minor.
Can verify that https://hub.docker.com/_/ubuntu/ with 22.04 works but 24.04 which comes with Python 3.12 has a bunch of Python issues.
Some can be easily worked around like https://stackoverflow.com/questions/75608323/how-do-i-solve-error-externally-managed-environment-every-time-i-use-pip-3 (or https://peps.python.org/pep-0668/ )
Edit to add: also libxml update seems to require system level packages libxml2-dev libxslt1-dev in 24.04 (possible regression?)
https://stackoverflow.com/questions/6504810/how-to-install-lxml-on-ubuntu
Our pinned cwltool/schema-salad version does indeed seem to have issues
Looks like we need to update to a cwltool version post 2023 August or October-ish if we want to be compatible with Python 3.12
https://github.com/common-workflow-language/cwltool/pull/1882 https://github.com/common-workflow-language/cwltool/pull/1919
Could add warning for user in 1.15.x hotfix, update cwltool in minor release
➤ David Steinberg commented:
Find out if there’s a workaround, if not throw up a warning saying don’t use this version of python yet.
➤ David Steinberg commented:
To review, try the directions on QA for setting up the CLI and cwltool https://qa.dockstore.org/quick-start ( https://qa.dockstore.org/quick-start|smart-link )
➤ David Steinberg commented:
This is part of the hotfix so should be tested in staging once that release is in testing.
➤ Steve Von Worley commented:
I tested our new cwltool installation instructions on MacOS. Using python 3.9, the last step failed as follows:
pipx runpip cwltool install -r requirements.txt ERROR: Ignored the following yanked versions: 2.6, 2.6.1 ERROR: Ignored the following versions that require a different python version: 3.3 Requires-Python >=3.10; 3.3rc0 Requires-Python >=3.10 ERROR: Could not find a version that satisfies the requirement networkx==3.3 (from versions: 0.34, 0.35, 0.35.1, 0.36, 0.37, 0.99, 1.0rc1, 1.0, 1.0.1, 1.1, 1.2rc1, 1.2, 1.3rc1, 1.3, 1.4rc1, 1.4, 1.5rc1, 1.5, 1.6rc1, 1.6, 1.7rc1, 1.7, 1.8rc1, 1.8, 1.8.1, 1.9rc1, 1.9, 1.9.1, 1.10rc2, 1.10, 1.11rc1, 1.11rc2, 1.11, 2.0, 2.1, 2.2rc1, 2.2, 2.3rc3, 2.3rc4, 2.3, 2.4rc1, 2.4rc2, 2.4, 2.5rc1, 2.5, 2.5.1, 2.6rc1, 2.6rc2, 2.6.2, 2.6.3, 2.7rc1, 2.7, 2.7.1, 2.8rc1, 2.8, 2.8.1rc1, 2.8.1, 2.8.2, 2.8.3, 2.8.4, 2.8.5, 2.8.6, 2.8.7, 2.8.8, 3.0b1, 3.0rc1, 3.0, 3.1rc0, 3.1, 3.2rc0, 3.2, 3.2.1) ERROR: No matching distribution found for networkx==3.3 '/Users/svonworl/.local/pipx/venvs/cwltool/bin/python -m pip install -r requirements.txt' failed python --version Python 3.9.13Looks like some of the requirements were yanked for Python versions prior to 3.10.
Python 3.9 reaches end-of-life in October 2025: https://devguide.python.org/versions/ ( https://devguide.python.org/versions/|smart-link )
So, in the instructions, we should probably note that Python 3.10+ is required to install cwltool.
Moving this back to “In Progress”.
➤ Denis Yuen commented:
Note: Steve and Ben seemed to have 3.9 versions (but Ben had possibly a different version via Homebrew, 3.12)