Packages should generate versioned dependencies on python

[fizbin]

I invoke dh_virtualenv in my package rule file with:

override_dh_virtualenv:
	dh_virtualenv --builtin-venv --python /usr/bin/python3 --skip-install --use-system-packages --upgrade-pip

This produces a package that works well on machines that have the same major version of python3 installed as the machine I built it on, but does not work on machines that have a different version of python3 installed. (i.e., build on a machine with python3.5, try to run on a machine with python3.8 -> failure)

The reason is that all my dependent packages end up installed inside a directory named something like /opt/venvs/mypackagename/lib/python3.8 where the last component is the python version.

In a case like this, the binary package that is produced should have a versioned dependency on python3 (>= 3.8), python3 (<< 3.9).

[jhermann]

You are the author of your control file, i.e. yes, do that. dh-virtualenv packages are by nature release-dependent. What could be an improvement to the project is documenting this.

[fizbin]

But the exact same source built on different machines will produce different packages.

E.g., if I build my package on an Ubuntu 16.04 LTS machine (xenial) I will get a package that will only work on a machine with python3 installed that is python3 (>= 3.5), python3 (<< 3.6).

If I build the exact same source on an Ubuntu 18.04 LTS (bionic) machine, I will get a package that will only work on a machine with python3 installed that is python3 (>= 3.6), python3 (<< 3.7).

If I build the exact same source on an Ubuntu 20.04 LTS (focal) machine, I will get a package that will only work on a machine with python3 installed that is python3 (>= 3.8), python3 (<< 3.9).

This is analogous to how building C programs on different hosts will produce dependencies on different versions of shared libraries.

At the very least dh_virtualenv should define a substvar that gives the bounds of acceptable python versions for the created virtual environment, so that I can add something like ${virtualenv:PyDepends} to my Depends line.

[fizbin]

For the record, here's the workaround I've developed at the moment:

override_dh_virtualenv:
	dh_virtualenv --builtin-venv --python /usr/bin/python3 --skip-install --use-system-packages --upgrade-pip
	# dh_virtualenv should tie us tightly to a specific python version, but it doesn't
	ls $(CURDIR)/debian/$(PACKAGE)$${DH_VIRTUALENV_INSTALL_ROOT}/$(PACKAGE)/lib | \
		perl -nle '/python((\d+)\.)((\d+\.)*)(\d+)$$/ && do {print "internal:PyDepends= python$$2 (>= $$1$$3$$5), python$$2 (<< $$1$$3",$$5+1,")"}' >> $(CURDIR)/debian/$(PACKAGE).substvars

And then I use ${internal:PyDepends} in my control file.

[nailor]

Thanks for reporting the issue.

First off, not working across Python versions is universally a known thing with virtualenv, not specific for a dh-virtualenv. This could of course be documented better. In general, when it comes to Debian and its derivates like Ubuntu, I would highly recommend building the packages on same OS as where those are going to be used to avoid issues

I like the idea of substvars being defined during installation, so definitely open for merging a PR if someone implements it.

Since the issue is not really an issue with dh-virtualenv, but a feature of virtualenvs in general, I'll be closing this issue.

[jhermann]

Regarding "build where you install", here's the hint again to use the contained Dockerfile to avoid polluting your build machines.

[fizbin]

For the record, the bug is not "the generated package doesn't work across python versions" but rather "because the generated package won't work across python versions, dh_virtualenv should generate versioned dependencies on python".

I'm not complaining that I can't take a package built on xenial and install and run it on a focal machine. I'm complaining that I can take a package built on xenial and dpkg is perfectly happy installing it on a focal machine. It should fail at the "package installation" phase, not at at the "run packaged program" phase.

[nailor]

Ah, sorry for misunderstanding!

I do agree and indeed the substvars would solve this issue. There is already some logic due to changes on Debian to record the major python version when building dh-virtualenv, something like that could be used to create substvars when building packages with it.

Just someone needs to implement all this 😜

On Tue 3. Nov 2020 at 23.52, Daniel Martin [email protected] wrote:

For the record, the bug is not "the generated package doesn't work across python versions" but rather "because the generated package won't work across python versions, dh_virtualenv should generate versioned dependencies on python".

I'm not complaining that I can't take a package built on xenial and install and run it on a focal machine. I'm complaining that I can take a package built on xenial and dpkg is perfectly happy installing it on a focal machine. It should fail at the "package installation" phase, not at at the "run packaged program" phase.

— You are receiving this because you modified the open/close state.

Reply to this email directly, view it on GitHub https://github.com/spotify/dh-virtualenv/issues/312#issuecomment-721391382, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAPRSESMWO5NU76XRDHSNDSOB3AJANCNFSM4TDCWVWQ .

--

• Jyrki

[WGH-]

Just for the record, how Synapse handles this problem:

https://github.com/matrix-org/synapse/blob/c7c84b81e3ec3d66f3a57a8d6ba3e58dd4c81ecc/debian/build_virtualenv#L117-L119

SNAKE=$(readlink -e /usr/bin/python3)
# add a dependency on the right version of python to substvars.
PYPKG=$(basename "$SNAKE")
echo "synapse:pydepends=$PYPKG" >> debian/matrix-synapse-py3.substvars

So the generated dependency is simply e.g. python3.9, not something complicated like python3 (>= 3.9), python3 (<< 3.10)