Old version of pip is installed into pack's virtualenv

[minsis]

SUMMARY

It seems that when the virtualenv's are built that some weird version of pip gets installed and I'm not sure where this is coming from. I even upgraded the system pip version to latest with no avail.

Because such an old version of pip gets installed it doesn't have support for some of the libs that get installed for things like st2client. Specifically orjson as it tries to build from source which requires rust to be installed.

STACKSTORM VERSION

st2 3.4.1, on Python 3.6.8

OS, environment, install method

HA cluster with two nodes and a controller.

Steps to reproduce the problem

Install a pack either manually or automatically. I'm not sure what the prereq's are. I have python 3.6 installed via yum. This installs an even older version of pip than what gets built in the virutalenv.

Expected Results

That the pip version from /opt/stackstorm/st2 gets installed and not some other version.

Actual Results

The install

<redacted> virtualenvs # /opt/stackstorm/st2/bin/virtualenv -p /opt/stackstorm/st2/bin/python --system-site-packages --verbose --no-download /opt/stackstorm/virtualenvs/zabbix -vvvv
Already using interpreter /opt/stackstorm/st2/bin/python
Using real prefix '/usr'
Creating /opt/stackstorm/virtualenvs/zabbix/lib/python3.6
This system uses lib64; symlinking lib64 to lib
Symlinking Python bootstrap modules
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib/python3.6/lib-dynload
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib/python3.6/config-3.6m-x86_64-linux-gnu
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/os.py
  Ignoring built-in bootstrap module: posix
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/posixpath.py
  Cannot import bootstrap module: nt
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/ntpath.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/genericpath.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/fnmatch.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/locale.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/encodings
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/codecs.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/stat.py
  Cannot import bootstrap module: UserDict
  File /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/lib-dynload/readline.cpython-36m-x86_64-linux-gnu.so already exists
  Cannot import bootstrap module: copy_reg
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/types.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/re.py
  Cannot import bootstrap module: sre
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/sre_parse.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/sre_constants.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/sre_compile.py
  File /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so already exists
  Cannot import bootstrap module: _abcoll
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/warnings.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/linecache.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/abc.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/io.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/_weakrefset.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/copyreg.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/tempfile.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/random.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/__future__.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/collections
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/keyword.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/tarfile.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/shutil.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/struct.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/copy.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/tokenize.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/token.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/functools.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/heapq.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/bisect.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/weakref.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/reprlib.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/base64.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/_dummy_thread.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/hashlib.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/hmac.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/imp.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/importlib
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/rlcompleter.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/operator.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/_collections_abc.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/_bootlocale.py
  Symlinking /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/enum.py
  No LICENSE.txt / LICENSE found in source
Creating /opt/stackstorm/virtualenvs/zabbix/lib/python3.6/site-packages
Writing /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/site.py
Writing /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/orig-prefix.txt
Creating parent directories for /opt/stackstorm/virtualenvs/zabbix/include
Symlinking /opt/stackstorm/virtualenvs/zabbix/include/python3.6m
File /opt/stackstorm/virtualenvs/zabbix/lib/python3.6/lib-dynload already exists
File /opt/stackstorm/virtualenvs/zabbix/lib/python3.6/config-3.6m-x86_64-linux-gnu already exists
Creating /opt/stackstorm/virtualenvs/zabbix/bin
New python executable in /opt/stackstorm/virtualenvs/zabbix/bin/python
Changed mode of /opt/stackstorm/virtualenvs/zabbix/bin/python to 0o755
Testing executable with /opt/stackstorm/virtualenvs/zabbix/bin/python -c "import sys;out=sys.stdout;getattr(out, "buffer", out).write(sys.prefix.encode("utf-8"))"
Got sys.prefix result: '/st2content/virtualenvs/zabbix'
Creating /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/distutils
Writing /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/distutils/__init__.py
Writing /opt/stackstorm/virtualenvs/zabbix/lib64/python3.6/distutils/distutils.cfg
Running command /opt/stackstorm/virtualenvs/zabbix/bin/python -m pip config list
Installing setuptools, pip, wheel...

  Running command /opt/stackstorm/virtualenvs/zabbix/bin/python - setuptools pip wheel
  Looking in links: /opt/stackstorm/st2/lib/python3.6/site-packages/virtualenv_support
  Collecting setuptools
  Collecting pip
  Collecting wheel
  Installing collected packages: setuptools, pip, wheel
  Successfully installed pip-19.1.1 setuptools-41.0.1 wheel-0.33.4
...Installing setuptools, pip, wheel...done.
Writing /opt/stackstorm/virtualenvs/zabbix/bin/activate
Writing /opt/stackstorm/virtualenvs/zabbix/bin/activate.fish
Writing /opt/stackstorm/virtualenvs/zabbix/bin/activate.csh
Writing /opt/stackstorm/virtualenvs/zabbix/bin/activate.ps1
Writing /opt/stackstorm/virtualenvs/zabbix/bin/activate_this.py
Writing /opt/stackstorm/virtualenvs/zabbix/bin/activate.xsh
Writing /opt/stackstorm/virtualenvs/zabbix/bin/python-config
Changed mode of /opt/stackstorm/virtualenvs/zabbix/bin/python-config to 0o755

This is the installed pip versions. I also upgraded the system pip3 to version 21.2.2 and still get 19.1.1 installed.

(st2) <redacted> virtualenvs # for i in $(ls); do $i/bin/pip --version; done
pip 19.1.1 from /opt/stackstorm/virtualenvs/ansible/lib/python3.6/site-packages/pip (python 3.6)
pip 19.1.1 from /opt/stackstorm/virtualenvs/bitbucket/lib/python3.6/site-packages/pip (python 3.6)
pip 19.1.1 from /opt/stackstorm/virtualenvs/chef/lib/python3.6/site-packages/pip (python 3.6)
pip 19.1.1 from /opt/stackstorm/virtualenvs/docker/lib/python3.6/site-packages/pip (python 3.6)
pip 19.1.1 from /opt/stackstorm/virtualenvs/jenkins/lib/python3.6/site-packages/pip (python 3.6)
pip 19.1.1 from /opt/stackstorm/virtualenvs/msexchange/lib/python3.6/site-packages/pip (python 3.6)
pip 19.1.1 from /opt/stackstorm/virtualenvs/opsgenie/lib/python3.6/site-packages/pip (python 3.6)
pip 19.1.1 from /opt/stackstorm/virtualenvs/servicenow/lib/python3.6/site-packages/pip (python 3.6)
pip 19.1.1 from /opt/stackstorm/virtualenvs/splunk/lib/python3.6/site-packages/pip (python 3.6)
pip 19.1.1 from /opt/stackstorm/virtualenvs/zabbix/lib/python3.6/site-packages/pip (python 3.6)
pip 19.1.1 from /opt/stackstorm/virtualenvs/zabbix/lib/python3.6/site-packages/pip (python 3.6)
(st2) <redacted> virtualenvs # pip3 --version
pip 20.0.2 from /opt/stackstorm/st2/lib/python3.6/site-packages/pip (python 3.6)
(st2) <redacted> virtualenvs # /usr/bin/pip3 --version
pip 9.0.3 from /usr/lib/python3.6/site-packages (python 3.6)

Thanks!

[amanda11]

Orjson I think was only used in st2 3.5.0.

St2 3.4.1 used pip 19.1.1 St2 3.5 uses pip 20.3.3.

I wonder if you are using the st2client version that was built for 3.5?

[minsis]

Locally I'm using what came with the st2 package.

The orjson install issue comes from a pack install failing, specifically Zabbix which installs the st2client no pinning. So its trying to the latest version of st2client while my local version is 3.4.1.

I guess this now makes sense was to why my package install is failing. But I still dont understand how 19.1.1 is being installed when the base virtualenv comes packaged with 20.3.3.

[amanda11]

@minsis If I remember correctly I think its because of the version of virtualenv itself. So virtualenv comes with a pip version bundled with it. So the version of virtualenv used in ST2 3.4.1 comes with pip 19.1.1, and we upgraded virtualenv in ST2 3.5, to use the version that bundles pip 20.3.3.

[minsis]

@amanda11 Upgrading to 3.5.0 fixed the issue. The new virtualenv's that get created are running 20.3.3 now which will now install the correct wheel for orjson instead of trying to build the core with rust. I guess some of the packs are not backwards compatible if they're requiring some sort of build from source for certain libraries. In this case if the pack needs to have st2client installed without pinning, its going to break.

[stale[bot]]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically marking is as stale. If this issue is not relevant or applicable anymore (problem has been fixed in a new version or similar), please close the issue or let us know so we can close it. On the contrary, if the issue is still relevant, there is nothing you need to do, but if you have any additional details or context which would help us when working on this issue, please include it as a comment to this issue.