Define process and implement venv migration for python minor release upgrades

[jdstrand]

This attempts to help define the TBD processes outlined in https://github.com/influxdata/influxdb/blob/main/README_processing_engine.md#how-will-influxdata-maintain-the-embedded-interpreter. It isn't strictly needed for GA since the code could be added at a later date. Updating to 3.13 prior to beta (see https://github.com/influxdata/influxdb/issues/26044) will provide even more breathing room. UPDATE: we updated to 3.13 before GA.

https://devguide.python.org/versions/ lists the support dates. Currently, 3.11 will receive support until 2027-10 and 3.13 until 2029-10.

Python uses MAJOR.MINOR.MICRO versioning. When a venv is created, it keeps a record of how the venv was created in /path/to/venv/pyvenv.cfg. Eg:

# Linux/OSX
$ cat /path/to/plugins/.venv/pyvenv.cfg 
home = /path/to/influxdb3/install/python/bin
include-system-site-packages = false
version = 3.11.11
executable = /path/to/influxdb3/install/python/bin/python3.11
command = /path/to/influxdb3/install/python/bin/python3 -m venv /path/to/plugins/.venv

# Windows
$ type \path\to\plugins\.venv\pyvenv.cfg
home = \path\to\influxdb3\install\python
include-system-site-packages = false
version = 3.11.11
executable = \path\to\influxdb3\install\python\python.exe
command = \path\to\influxdb3\install\python\python.exe -m venv \path\to\plugins\.venv

The details are discussed in https://peps.python.org/pep-0405/, but importantly, influxdb3 should pay attention to the version field (NOT executable since it doesn't encode the version on windows) and the in use python runtime. When MAJOR.MINOR is the same, the venv is expected to work and we shouldn't have to do anything, but when different, we need to take action.

A straw person on how to handle this might be:

1. on initial creation and install package, run python3 -n pip freeze > requirements.txt
   • how to handle/detect manual pip installs? Perhaps unconditionally run python3 -n pip freeze > requirements.txt prior to venv activation?
2. on upgrade from different minors (eg, 3.11.11 to 3.11.12), do nothing
   • perhaps we want to log at INFO or DEBUG?
3. It's possible to do an in place upgrade from different minors (eg, 3.11 to 3.13). Eg:

   # pseudocode
   warn!("Detected new python version, backing up .venv to .venv-3.11.zip and migrating to 3.13")
   mv /path/to/plugins/.venv /path/to/plugins/.venv-3.11  # 3.11 calculated from 'version' in pyvenv.cfg
   python3 -m venv /path/to/plugins/.venv   # using new python3
   cp /path/to/plugins/.venv-3.11/requirements.txt /path/to/plugins/.venv
   python3 -m pip -r /path/to/plugins/.venv-3.11/requirements.txt   # using new python3
   zip /path/to/plugins/.venv-3.11
   warn!("Migration complete. Please test to ensure functionality.")
   

4. We should have clear instructions for how users can test functionality prior to upgrades. Eg: a. install new influxdb3 to a different location b. copy the production 'plugins' directory (and venv if separate) to a new location c. point the new influxdb3 serve invocation to the new 'plugins' directory (and venv if separate) d. test functionality, adjusting scripts and requirements.txt as necessary e. if testing required no changes, perform an upgrade like normal; otherwise, shutdown influxdb3, perform upgrade, copy updated scripts to 'plugins' and copy updated requirements.txt to venv, perform upgrade

The details of '3' and '4' of course may need to be updated from the above, but that is the general idea. Details for handling alternative venvs (ie, --virtual-env-location) intentionally omitted.

[pauldix]

Is this still an issue? Is there something short and actionable for us to do here? I'm looking for a summary.

[jdstrand]

It is still needed, yes, since we can't move to a new python minor release without breaking users (an existing venv will point to the wrong python minor version). The good news is, micro versions are not a problem and we upgraded to 3.13 before GA, which is supported until 2029 (so upstream security and bug fixes in 3.13.x are safe to apply).

I'll note that https://github.com/influxdata/influxdb/issues/26062#issuecomment-2684966769 discusses a related issue when reusing the plugin directory. Considering the 2029 support timeline, it might be tempting to kick the can on this issue until much later, but the plugin dir reuse issue is almost certainly going to need to be fixed sooner than that, and the fix for it should consider this issue.

[jdstrand]

actionable for us

Someone needs to investigate the best path forward and implement it. My strawperson was given for inspiration for whoever picks that up.