Improve Python install

[mgorny]

Besides installing cvise to site-packages, we need to get the modules byte-compiled. Now, that's a non-trivial task and CMake is horrible at it, so it'd be best to actually use distutils/setuptools for that. Ideally, the package could be split into two, one containing common C/C++ tooling that could shared between cvise and creduce, and the other containing the Python driver.

Why byte-compiling is hard? The exact commands used depend on Python interpreter and version used. There are different commands for Python <3.5, PyPy(2), Python 3 (and PyPy3) >=3.5, Jython… and these are the implementations we used. Even Python upstream recently gets it wrong, and we have to patch Python for it.

The 'fallback' code we have for it: https://gitweb.gentoo.org/repo/gentoo.git/tree/eclass/python-utils-r1.eclass#n595

I don't think you want to repeat it verbatim in CMakeLists.txt and maintain it going forward.

[marxin]

Besides installing cvise to site-packages, we need to get the modules byte-compiled. Now, that's a non-trivial task and CMake is horrible at it, so it'd be best to actually use distutils/setuptools for that. Ideally, the package could be split into two, one containing common C/C++ tooling that could shared between cvise and creduce, and the other containing the Python driver.

It would be ideal, but now there's some divergence, partially because I cherry-picked some long-lasting pull requests of creduce.

Why byte-compiling is hard? The exact commands used depend on Python interpreter and version used. There are different commands for Python <3.5, PyPy(2), Python 3 (and PyPy3) >=3.5, Jython… and these are the implementations we used. Even Python upstream recently gets it wrong, and we have to patch Python for it.

It seems to me as a nice optimization, but in case of C-Vise one expects the script running for quite some time, so caching is probably not a crucial need.

The 'fallback' code we have for it: https://gitweb.gentoo.org/repo/gentoo.git/tree/eclass/python-utils-r1.eclass#n595

I don't think you want to repeat it verbatim in CMakeLists.txt and maintain it going forward.

Maybe we can create a cmake function that will both install .py file and it's cached file? Thanks for the idea!

[mgorny]

Besides installing cvise to site-packages, we need to get the modules byte-compiled. Now, that's a non-trivial task and CMake is horrible at it, so it'd be best to actually use distutils/setuptools for that. Ideally, the package could be split into two, one containing common C/C++ tooling that could shared between cvise and creduce, and the other containing the Python driver.

It would be ideal, but now there's some divergence, partially because I cherry-picked some long-lasting pull requests of creduce.

But do these break backwards compatibility? I was thinking of taking your version and using it in creduce.

Why byte-compiling is hard? The exact commands used depend on Python interpreter and version used. There are different commands for Python <3.5, PyPy(2), Python 3 (and PyPy3) >=3.5, Jython… and these are the implementations we used. Even Python upstream recently gets it wrong, and we have to patch Python for it.

It seems to me as a nice optimization, but in case of C-Vise one expects the script running for quite some time, so caching is probably not a crucial need.

The main problem is not really whether the cache exists but that Python will try to write one if it doesn't. This means orphan files, and we have QA checks in place to detect that.

The 'fallback' code we have for it: https://gitweb.gentoo.org/repo/gentoo.git/tree/eclass/python-utils-r1.eclass#n595 I don't think you want to repeat it verbatim in CMakeLists.txt and maintain it going forward.

Maybe we can create a cmake function that will both install .py file and it's cached file?

I think the usual approach is to byte-compile all files after they're installed.

Though I'd still lean towards using distutils for that. You can always make CMake call setup.py with proper arguments, roughly ${PYTHON_EXECUTABLE} setup.py install --prefix=${CMAKE_INSTALL_PREFIX} --root=${DESTDIR} (sorry, I don't recall exactly how to substitute DESTDIR there).

[marxin]

But do these break backwards compatibility? I was thinking of taking your version and using it in creduce.

No, there's no breakage of the compatibility. Feel free to use C-Vise's binaries for creduce.