feature: shims as .exe files

[db4]

trafficstars

Current implementation of shims as .bat files breaks some existing use cases. E.g. CMake wrapper for conan contains the following code fragment to call Conan:

        execute_process(COMMAND ${CONAN_CMD} ${conan_args}
                        RESULT_VARIABLE return_code
                        OUTPUT_VARIABLE conan_output
                        ERROR_VARIABLE conan_output
                        WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})

normally CONAN_CMD points to <python dir>\Scripts\conan.exe, but with pyenv-win it turns to shims\conan.bat, so execute_process() fails (it doesn't use cmd.exe to run commands)

Maybe pyenv-win could use Chocolatey Shim Generator to work around this? Although it's not OSS, they're ready to grant a license:

I want to use shimgen outside of Chocolatey.

If your project is FOSS, please contact us for a grant of a free license to do so

[db4]

The following proof of concept works for me so far:

--- pyenv-lib.vbs.orig	2021-09-16 15:40:10.000000000 +0300
+++ pyenv-lib.vbs	2022-02-16 11:15:15.096235800 +0300
@@ -256,14 +256,18 @@
 Sub WriteWinScript(baseName)
     ' WScript.echo "kkotari: pyenv-lib.vbs write win script..!"
     Dim filespec
-    filespec = strDirShims &"\"& baseName &".bat"
+    Dim objexec
+    Dim shimgen
+    Dim cmd
+    shimgen = objws.Environment("Process")("ChocolateyInstall")&"\tools\shimgen.exe"
+    cmd = objws.Environment("Process")("COMSPEC")
+    filespec = strDirShims &"\"& baseName &".exe"
     If Not objfs.FileExists(filespec) Then
-        With objfs.CreateTextFile(filespec)
-            .WriteLine("@echo off")
-            .WriteLine("chcp 1250 > NUL")
-            .WriteLine("call pyenv exec %~n0 %*")
-            .Close
-        End With
+        Set objexec = objws.Exec(shimgen & " -o " & filespec & " -p " & cmd & " -c ""/C pyenv exec " & baseName & """")
+        Do
+            WScript.StdOut.WriteLine(objExec.StdOut.ReadLine())
+        Loop While Not objexec.Stdout.atEndOfStream
+        WScript.StdOut.WriteLine(objexec.StdOut.ReadAll)
     End If
 End Sub

[kirankotari]

@db4 Hey it's good to create a pull request for it. Please go ahead.

[bkeryan]

Is it necessary to dynamically generate shim EXEs? A single, precompiled shim EXE could use argv[0] to identify which command to call.

[db4]

Is it necessary to dynamically generate shim EXEs? A single, precompiled shim EXE could use argv[0] to identify which command to call.

How this identification could work? Maintain the map in a separate file (one for every shim directory)? Yes, that's possible, but IMHO harder to administer and requires some efforts to implement (Chocolatey Shim Generator is already there)

[bkeryan]

The existing batch file implementation already uses this technique. %0 is argv[0]. %~n0 extracts the command name, omitting the drive letter, directory, and file extension. pyenv exec does the rest.

In C on Windows, you can use _splitpath_s to extract the command name:

char command[MAX_PATH] = "";
errno_t error = _splitpath_s(argv[0], NULL, 0, NULL, 0, command, sizeof(command), NULL, 0);

The result is a command name that you can pass to pyenv exec along with argv[1] through argv[argc - 1]. For example:

argv[0]	command
python	python
python3.exe	python3
C:\Users\Username\.pyenv\pyenv-win\shims\pip.exe	pip
..\..\.pyenv\pyenv-win\shims\pythonw.exe	pythonw

[db4]

@bkeryan I like your approach. I've created a helper exe in C and it seems to work with pyenv without an issue. The only problem is how to integrate it into pyenv-win distribution? C sources need Visual Studio to build them...

[db4]

No, still not entirely acceptable. FindPython3.cmake detects python the following way:

  if (_${_PYTHON_PREFIX}_EXECUTABLE AND NOT CMAKE_CROSSCOMPILING)
    if (NAME STREQUAL "PREFIX")
      execute_process (COMMAND ${_${_PYTHON_PREFIX}_INTERPRETER_LAUNCHER} "${_${_PYTHON_PREFIX}_EXECUTABLE}" -c "import sys\ntry:\n   from distutils import sysconfig\n   sys.stdout.write(';'.join([sysconfig.PREFIX,sysconfig.EXEC_PREFIX,sysconfig.BASE_EXEC_PREFIX]))\nexcept Exception:\n   import sysconfig\n   sys.stdout.write(';'.join([sysconfig.get_config_var('base') or '', sysconfig.get_config_var('installed_base') or '']))"
                       RESULT_VARIABLE _result
                       OUTPUT_VARIABLE _values
                       ERROR_QUIET
                       OUTPUT_STRIP_TRAILING_WHITESPACE)

Note these newline characters. I don't know a way to correctly escape them for pyenv.bat. Probably the only chance is to re-implement pyenv.bat exec logic in C.

[Cologler]

I created https://github.com/Cologler/exe2ps1-rust for this and used it for a couple of months. The problem is it will be removed by pyenv after calling pip install ....

[db4]

@Cologler does FindPython3.cmake from CMake distribution work correctly with your python.exe?

[dopry]

@bkeryan @db4 any way you could make a PR with your code or bundled 'shim.exe'? One possible route for inclusion that would reduce the packaging overhead would be a pre-bundled shim.exe, that way it doesn't need to be compiled in the pyenv0win build process? This bit me with pyenv shell on windows not picking up the correct shell.

[kirankotari]

Hi All, we need generic way to create the file.exe, not everyone uses Chocolatey in Windows. Second, we can't go symbolic link to the exe file - which will not work microsoft link Looks like shortcuts might work Ref. Link

[db4]

@dopry I've already explained that just providing shim.exe of any kind wouldn't help much. The problem is pyenv.bat that's always involved in the process. To make the whole thing work correctly one would need to implement pyenv.bat in a more adequate language (that allows to pass command line parameters in an array without quoting)

[dopry]

@db4 oh I didn't fully grok that while reading the issue thread. It looks like pyenv.bat mostly passes calls through to lib-exec/pyenv.vbs, with the exception of help modules and a few other tasks. So would compiling that vbs to a .exe and internalizing help be another a possible solution? Alternatively using python would be nice. I suspect there are more python developers than bat, ps, or even vbs in the community who would be willing to support the project.

[kirankotari]

@dopry @db4 @Cologler @bkeryan here is the PR #463 test this let us know.

[dopry]

@kirankotari may work for conan. My use case is pipenv shell preserving my shell. I think this would be one step toward resolving that issue. It didn't work for pipenv shell in my tests. I don't want to hijack the conan issue. I'll wait for it to get resolved and then test and see if I can get pipenv playing nice once it's done.

[ddspj23]

lnk files don't seem to be parsed and run very well by various shells. I had to add the.lnk suffix to make it work when I tried it in powershell, and for some reason, parameter passing didn't work. In some other shells the lnk file is not recognized as an executable link at all. Back to the bat file, in addition to the problems mentioned before, when running the script obtained from the network, because there is no "call" command before shim-commands, they can not be executed correctly. The current version creates bat for exe files and lnk for others, but I think creating shims in exe format using the method chocolatey used for exe files seems to be the better choice in all respects.

[kirankotari]

@ddspj23 got it, for now I can open the both options and let's keep this thread open to check why in few cases its working and in few cases and shell it's not.

[ddspj23]

lnk files don't seem to be parsed and run very well by various shells. I had to add the.lnk suffix to make it work when I tried it in powershell, and for some reason, parameter passing didn't work. In some other shells the lnk file is not recognized as an executable link at all. Back to the bat file, in addition to the problems mentioned before, when running the script obtained from the network, because there is no "call" command before shim-commands, they can not be executed correctly. The current version creates bat for exe files and lnk for others, but I think creating shims in exe format using the method chocolatey used for exe files seems to be the better choice in all respects.

Also: Using exe files as shims also solves the problem of not responding properly to calls like python.exe.

[ddspj23]

@kirankotari Ok, thank you for your continued selfless dedication to this project.

[db4]

I've finally implemented shim.exe in Python (using pyinstaller). As @bkeryan suggested, no need to generate it dynamically, one EXE is enough. Here it is:

import os
import shutil
import subprocess
import sys

if __name__ == "__main__":
    if not getattr(sys, 'frozen', False):
        sys.stderr.write("This should be run as a frozen exe\n")
        sys.exit(1)

    pyenv_root = os.path.dirname(os.path.dirname(sys.executable))

    pyenv_vbs = os.path.join(pyenv_root, "libexec", "pyenv.vbs")
    result = subprocess.run(["cscript", "/nologo", pyenv_vbs, "vname"], check=True, capture_output=True, text=True)
    ver = result.stdout.strip()
    bin_dir = os.path.join(pyenv_root, "versions", ver)
    scripts_dir = os.path.join(bin_dir, "Scripts")

    python_shim = os.path.normcase(os.path.join(pyenv_root, "shims", "python.exe"))
    python_bin = os.path.normcase(os.path.join(pyenv_root, "versions", ver, "python.exe"))
    python_in_path = shutil.which("python.exe")
    if python_in_path:
        python_in_path = os.path.normcase(python_in_path)
    if python_in_path and python_in_path not in [python_shim, python_bin]:
        sys.stderr.write(f"Wrong {python_in_path} is in the PATH\n")
        sys.exit(1)

    exe = os.path.basename(sys.executable)
    exe_path = None
    for dir in [bin_dir, scripts_dir]:
        path = os.path.join(dir, exe)
        if os.path.exists(path):
            exe_path = path
    if not exe_path:
        sys.stderr.write(f"{exe} is not found")
        sys.exit(1)

    env_copy = os.environ.copy()
    env_copy["PATH"] = os.pathsep.join([bin_dir, scripts_dir, env_copy["PATH"]])
    result = subprocess.run([exe_path] + sys.argv[1:], env=env_copy)
    sys.exit(result.returncode)

This works for me and seems to solve all the problems described above. I could create a PR but don't quite understand how to integrate pyinstaller-created shim.exe into pyenv-win distribution.

BTW, it's better not to copy shim.exe, but just symlink it if possible. I currently do this as follows:

' pyenv - bin - windows
Sub WriteWinScript(baseName)
    ' WScript.echo "kkotari: pyenv-lib.vbs write win script..!"
    Dim shim
    Dim filespec
    shim = strDirLibs &"\libs\shim.exe"
    filespec = strDirShims &"\"& baseName &".exe"
    If Not objfs.FileExists(filespec) Then
        objws.Run "cmd /C mklink " & filespec & " " & shim, 0
    End If
End Sub

Maybe anyone know a native way to create a symlink in VB Script?

[bkeryan]

@db4 Are you building this as a one-folder or one-file EXE? One-file EXEs extract files to a temp directory and spawn a child process, so one-folder EXEs would have a lower impact on command execution time.

[db4]

@db4 Are you building this as a one-folder or one-file EXE? One-file EXEs extract files to a temp directory and spawn a child process, so one-folder EXEs would have a lower impact on command execution time.

@bkeryan I tried one-file EXE. I know that it extracts files to a temp directory, but it takes a fraction of a second, so seems acceptable. The size of .exe is a bigger issue: it's about 4MB so the whole shims directory is > 400Mb (if you copy shims, not symlink them). So I'm thinking of reimplementing the same logic in C.