Unable to select Python 3.10 interpreter

[matheger]

Problem Description

Choosing a Python 3.10.0 environment in "Preferences" -> "Python interpreter" pops up an error saying, "Invalid file path: (path to python.exe)".

Looks the exact same as #15359.

Works with any of the other 3.9.7 environments I have currently installed. Downgrading the bad one to 3.9 also works.

I'm attaching an anonymized --debug-info verbose log snippet from a failed attempt to switch to the 3.10 interpreter, and a working one for comparison (with timestamps removed):

switch-failed.txt switch-ok.txt

What steps reproduce the problem?

1. Create a new 3.10 environment: conda create -n py310 python=3.10
2. Select interpreter in Preferences dialog
3. boom

Versions

• Spyder version: 5.1.5 None
• Python version: 3.9.7 64-bit
• Qt version: 5.9.7
• PyQt5 version: 5.9.2
• Operating System: Windows 10

[ccordoba12]

Hey @matheger, thanks for reporting. I can't reproduce this on Linux.

@dalthviz, could you check if you get this error on Windows?

[dalthviz]

I was able to reproduce this on Windows. However for me before the invalid path error dialog I got a System error dialog related to a missing .dll (zlib.dll):

I also tried creating an env with Python 3.10 but using the conda-forge channel (which install Python 3.10.2) and there the interpreter is recognized. If I install the corresponding spyder-kernels version compatible with Spyder 5.1.5 from the conda-forge channel in the py310 env I'm able to start a console:

So to create the env I used:

conda create -n py310 python=3.10 -c conda-forge
conda activate py310
conda install -c conda-forge spyder-kernels=2.1.3

[matheger]

Interesting. With the 3.10.2 kernel from conda-forge, it works for me as well. It might be just that one binary that makes it run into trouble. (Having spyder-kernels installed also makes no difference in whether or not I can select those kernels in the settings.)

The thing that stood out to me in the log file is that there's not much more going on than some attempts at checking for languages and encodings. The only real difference I can see is that the best bet for the encoding is "Turkish" for the failing 3.10 kernel and "Greek" for the working 3.9.7.

Seems like quite a bit of fringe issue overall, but it's happened before so maybe it's worth finding out what is going on for the future. I'm happy to help if there's anything else I can do.

[matheger]

I've found the problem. The spyder.utils.programs.check_python_help method checks for a valid python interpreter by essentially running

subprocess.Popen([<path to interpreter>, '-c', 'import this'], env=<some dict of variables>, ...)

and making sure that it yields the correct first four lines of the Zen of Python. I tested this Popen call in a standalone interpreter outside of Spyder, and it only works with Python 3.10.0 if I don't pass the env parameter. If I do, then it complains about not being able to find zlib.dll (similarly to what @dalthviz reported) and the output comes back empty, thus the "zen check" fails. No idea why it fails quietly for me if I do it from inside Spyder but in any case, this is where it breaks. Also no surprise that this doesn't happen on Linux.

For me, the dict handed to env is simply {'SYSTEMROOT': 'C:\\WINDOWS', 'SYSTEMDRIVE': 'C:'}. I have no idea why that makes it go bad. In fact, it doesn't seem to matter what the entries in the dict actually are, as long as it's non-empty.

[dalthviz]

Thank you for checking into this @matheger ! Is quite strange for that to happen with Python 3.10.0 (maybe a bug in that Python version?). Thank you again for letting us know :+1:

[matheger]

Okay, this is weird, but I shuffled a couple 3.10 subversions around because I wanted to know which ones would work and now I can't reproduce the Popen error from a standalone console anymore. Works fine with every version of it as long as I give it a correct 'SYSTEMROOT' env variable.

At the back of my head I was also wondering, wouldn't it be more robust to base the interpreter check around the file hashes for the python executables? The "zen check" works but seems very hacky.

[ccordoba12]

wouldn't it be more robust to base the interpreter check around the file hashes for the python executables?

How many of those would we need to make the validation pass? I mean, you can install Python through Python.org, Anaconda, Conda-forge, Pyenv, Linux distros, Mac Ports, Howebrew, etc etc. And then, every minor version (10 in the case of Python 3.7!) probably has a different hash as well.

The "zen check" works but seems very hacky.

The idea is to verify that the Python executable can actually run code. If you have other ideas for such a check, we're welcome to hear them.

[matheger]

The idea is to verify that the Python executable can actually run code. If you have other ideas for such a check, we're welcome to hear them.

Well, if we had a conclusive list of valid interpreter hashes, then matching a hash would guarantee that the executable can run code. I don't know much about the binary files that are supplied from different sources, though. Would they just be repackaged versions of "official" binaries from python.org? I could look into that if you think it might be a worthwhile strategy.

[ccordoba12]

Would they just be repackaged versions of "official" binaries from python.org?

They aren't. And with Pyenv you can compile different Python versions locally.

I could look into that if you think it might be a worthwhile strategy.

I think it's untenable, so I wouldn't invest time on it.

[callegar]

Seeing the same trying to use a virtual environment where python is actually pyston that should be the same as python 3.8.12.

I see two different issues here:

1. Not recognizing as usable a python interpreter that should be valid;
2. Providing a very misleading error. This is not an invalid path. The path is perfectly valid. At best the message should be Unsuitable interpreter for spyder at given path or something like that.

[cvergari]

Seeing the same issue with Spyder 5.3.3 and python 3.10.9 in the base and python 3.6.15 in my environment! The problem is that python 3.10.2 is not compatible with the packages in my environment (nor is 3.9), so I am unfortunately stuck.

Everything works in other environments with python 3.9.16 and 3.11.0.

[ccordoba12]

@cvergari, so the problem is that Spyder can't detect your Python 3.6 environment?

[cvergari]

Yes, I get the "Invalid file path: (path to python.exe)" error. But I also realized that even if this worked, I would have some trouble using spyder-kernels 2.4 with python 3.6...

[cvergari]

I got the "Invalid file path: (path to python.exe)" error when pointing to my environment's python.exe. But the problem seems resolved with Spyder 5.4! I updated Spyder in my base, and now I can select python 3.6 in my environment. Now I have cannot use spyder-kernels 2.4 with python 3.6...but that's a whole other issue.