First-run experience failure: `rye run hello` after `rye init` + `rye sync` -- results in "ModuleNotFoundError: no module named 'xyz' ..."

[amontalenti]

Steps to Reproduce

1. I created a basic project with rye init.
2. I used the [email protected] (but also tried with [email protected]) -- these are the the only two toolchains installed
3. rye run python works to pull up Python 3.11.7
4. rye sync works without error
5. rye run lists hello as a valid run argument
6. but, rye run hello results in error

Expected Result

I would expect the hello() function to get called, which I imagine would just succeed since the default implementation of that function simply returns a string.

Actual Result

Traceback (most recent call last):
  File "/home/am/repos/xyz/.venv/bin/hello", line 5, in <module>
    from xyz import hello
ModuleNotFoundError: No module named 'xyz'

I should mention that I can "fix" this by doing the following from the root of the project:

cp -R src/xyz .venv/lib/python3.11/site-packages

At that point, rye run hello succeeds and returns:

❯ cp -R src/xyz .venv/lib/python3.11/site-packages/
❯ rye run hello
Hello from xyz!

But obviously I shouldn't have to manually copy the project source into the .venv directory for rye run to work.

I thought maybe this had something to do with me using pyenv before rye, but I tried the same from a shell where I removed pyenv from the $PATH and also removed it from zsh plugins, and yet, still, rye behaved same way.

Version Info

rye 0.27.0
commit: 0.27.0 (43ee4fce0 2024-02-26)
platform: linux (x86_64)
self-python: [email protected]
symlink support: true
uv enabled: true

Stacktrace

RUST_BACKTRACE=1 has no effect on rye run, so I guess it's not crashing.

[dsp]

Thank you for reporting this.

I cannot reproduce this for now:

$ rye init xyz
$ cd xyz
$ rye sync
$ rye run
python
python3
python3.12

Did pyproject.toml already exist or did you add a new project.scripts entry? What's your pyproject.toml look like? Can you pastei t?

There was a change in v0.26.0 that changed the templates for the projects, which should result in a new rye init not even producing a hello script in rye run.

[T-256]

Try activate your virtual environment: source .venv/bin/activate

[amontalenti]

Re: @T-256's comment, same result after running venv activation.

Re: @dsp's comment, here's my pyproject.toml:

[project]                                                                                                          
name = "xyz"                                                                                                       
version = "0.2.2"                                                                                                  
description = "xyz placeholder project (under development)"                                                        
authors = [                                                                                                        
    { name = "Andrew Montalenti", email = "[email protected]" }                                                  
]                                                                                                                  
dependencies = [                                                                                                   
]                                                                                                                  
readme = "README.md"                                                                                               
requires-python = ">= 3.8"                                                                                         
                                                                                                                   
[project.scripts]                                                                                                  
hello = "xyz:hello"                                                                                                
                                                                                                                   
[build-system]                                                                                                     
requires = ["hatchling"]                                                                                           
build-backend = "hatchling.build"                                                                                  
                                                                                                                   
[tool.rye]                                                                                                         
managed = true                                                                                                     
dev-dependencies = []                                                                                              
                                                                                                                   
[tool.hatch.metadata]                                                                                              
allow-direct-references = true                                                                                     
                                                                                                                   
[tool.hatch.build.targets.wheel]                                                                                   
packages = ["src/xyz"]

[amontalenti]

Also re: @dsp's comment, given that he mentioned that rye init templates changed and I did adopt rye a few versions back, I decided to follow his own re-production steps. This is what happened. (Notice that import xyz failed in the python shell at the end.)

❯ rye init xyz
success: Initialized project in /home/am/repos/xyz
  Run `rye sync` to get started
❯ cd xyz
❯ rye sync
Initializing new virtualenv in /home/am/repos/xyz/.venv
Python version: [email protected]
Generating production lockfile: /home/am/repos/xyz/requirements.lock
Generating dev lockfile: /home/am/repos/xyz/requirements-dev.lock
Installing dependencies
   Built file:///home/am/repos/xyz                                                                                 Built 1 editable in 212ms
Installed 1 package in 0ms
 + xyz==0.1.0 (from file:///home/am/repos/xyz)
Done!
❯ rye run
python
python3
python3.12
❯ rye run python
Cannot read termcap database;
using dumb terminal settings.
Python 3.12.1 (main, Jan  8 2024, 05:57:25) [Clang 17.0.6 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import xyz
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'xyz'
>>> 

[cnpryer]

FWIW I could not repro the latest example

@cnpryer ➜ /workspaces $ rye init xyz
success: Initialized project in /workspaces/xyz
  Run `rye sync` to get started
@cnpryer ➜ /workspaces $ cd xyz
@cnpryer ➜ /workspaces/xyz (main) $ rye sync
Initializing new virtualenv in /workspaces/xyz/.venv
Python version: [email protected]
Generating production lockfile: /workspaces/xyz/requirements.lock
Generating dev lockfile: /workspaces/xyz/requirements-dev.lock
Installing dependencies
   Built file:///workspaces/xyz                                                                                                                                   Built 1 editable in 257ms
Installed 1 package in 0ms
 + xyz==0.1.0 (from file:///workspaces/xyz)
Done!
@cnpryer ➜ /workspaces/xyz (main) $ rye run python
Python 3.12.2 (main, Feb 25 2024, 04:38:01) [Clang 17.0.6 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import xyz
>>> 
@cnpryer ➜ /workspaces/xyz (main) $ rye --version
rye 0.27.0
commit: 0.27.0 (43ee4fce0 2024-02-26)
platform: linux (x86_64)
self-python: [email protected]
symlink support: true
uv enabled: true

[amontalenti]

I imagine it must be something about my environment (specific zsh profile, Ubuntu 20.04 LTS, etc.) ... What other environmental details can I provide to figure it out?

[dsp]

@amontalenti Yes, that's a good guess. Do you have $PYTHONPATH or something similar set (you can check using env). It might be worht trying: env -u PYTHONHOME -u PYTHONPATH rye run python.

[amontalenti]

@dsp Good idea, but didn't seem to help:

❯ cd ~/repos/xyz
❯ env -u PYTHONHOME -u PYTHONPATH rye run python
Cannot read termcap database;
using dumb terminal settings.
Python 3.12.1 (main, Jan  8 2024, 05:57:25) [Clang 17.0.6 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import xyz
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'xyz'

[amontalenti]

I also tried simplifying my PATH, with same result. Just the rye shims + system paths.

❯ echo $PATH
/home/am/.rye/shims:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

[amontalenti]

What would probably help me is to understand, based on this contents inside site-packages:

❯ ls .venv/lib/python3.12/site-packages
_tcl-init.pth  _virtualenv.pth  _virtualenv.py  xyz-0.1.0.dist-info  _xyz.pth

How is import xyz supposed to work? That is, how is it supposed to find the xyz package given that no folder named xyz is put into the .venv folder by rye sync? What magic am I missing?

[smaxad47]

I have the same problem, on Windows.

[dsp]

What would probably help me is to understand, based on this contents inside site-packages:

❯ ls .venv/lib/python3.12/site-packages
_tcl-init.pth  _virtualenv.pth  _virtualenv.py  xyz-0.1.0.dist-info  _xyz.pth

How is import xyz supposed to work? That is, how is it supposed to find the xyz package given that no folder named xyz is put into the .venv folder by rye sync? What magic am I missing?

The c6ntent 6f _xyz.pth will be added to sys.path (see https://docs.python.org/3/library/site.html). xyz-0.1.0.dist-info should contain a package that will have a direct_url.json pointing to your src/xyz directory on disk.

One thing you could try @amontalenti is to see what rye run python3 and then print(sys.path) looks like. I am also on discord, feel free to reach me during european work hours.

[amontalenti]

Thanks @dsp, these are good tips.

When I cat the _xyz.path file, it's empty:

❯ cat .venv/lib/python3.12/site-packages/_xyz.pth

I do have the direct_url.json file with the pointer to the right place, though.

❯ cat .venv/lib/python3.12/site-packages/xyz-0.1.0.dist-info/direct_url.json
{"url":"file:///home/am/repos/xyz","dir_info":{"editable":true}}                                                                                                        

As for the sys.path in the python3 interpreter itself:

❯ rye run python3
Python 3.12.1 (main, Jan  8 2024, 05:57:25) [Clang 17.0.6 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pprint import pprint
>>> import sys
>>> pprint(sys.path)
['',
 '/home/am/.rye/py/[email protected]/install/lib/python312.zip',
 '/home/am/.rye/py/[email protected]/install/lib/python3.12',
 '/home/am/.rye/py/[email protected]/install/lib/python3.12/lib-dynload',
 '/home/am/repos/xyz/.venv/lib/python3.12/site-packages']
>>> 

... but yet:

>>> import xyz
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'xyz'

[amontalenti]

I also stole an idea from another GitHub issue, which is to run the python3 interpreter under a heavily verbose mode, which will show where it tries to do module lookup upon import. Here are those results:

❯ rye run python3 -vvvv
[... elided lots of output ...]
>>> import xyz
# trying /home/am/repos/xyz/xyz.cpython-312-x86_64-linux-gnu.so
# trying /home/am/repos/xyz/xyz.abi3.so
# trying /home/am/repos/xyz/xyz.so
# trying /home/am/repos/xyz/xyz.py
# trying /home/am/repos/xyz/xyz.pyc
# trying /home/am/.rye/py/[email protected]/install/lib/python3.12/xyz.cpython-312-x86_64-linux-gnu.so
# trying /home/am/.rye/py/[email protected]/install/lib/python3.12/xyz.abi3.so
# trying /home/am/.rye/py/[email protected]/install/lib/python3.12/xyz.so
# trying /home/am/.rye/py/[email protected]/install/lib/python3.12/xyz.py
# trying /home/am/.rye/py/[email protected]/install/lib/python3.12/xyz.pyc
# trying /home/am/.rye/py/[email protected]/install/lib/python3.12/lib-dynload/xyz.cpython-312-x86_64-linux-gnu.so
# trying /home/am/.rye/py/[email protected]/install/lib/python3.12/lib-dynload/xyz.abi3.so
# trying /home/am/.rye/py/[email protected]/install/lib/python3.12/lib-dynload/xyz.so
# trying /home/am/.rye/py/[email protected]/install/lib/python3.12/lib-dynload/xyz.py
# trying /home/am/.rye/py/[email protected]/install/lib/python3.12/lib-dynload/xyz.pyc
# trying /home/am/repos/xyz/.venv/lib/python3.12/site-packages/xyz.cpython-312-x86_64-linux-gnu.so
# trying /home/am/repos/xyz/.venv/lib/python3.12/site-packages/xyz.abi3.so
# trying /home/am/repos/xyz/.venv/lib/python3.12/site-packages/xyz.so
# trying /home/am/repos/xyz/.venv/lib/python3.12/site-packages/xyz.py
# trying /home/am/repos/xyz/.venv/lib/python3.12/site-packages/xyz.pyc
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1324, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'xyz'
>>> 

[amontalenti]

One more pretty whacky result.

Out of sheer curiosity, I tried installing pip into the rye-managed venv. This is because I wanted to inspect the output of pip list:

❯ rye run python3 -m ensurepip
[...]
❯ rye run python3 -m pip install --upgrade pip
[...]
❯ rye run python3 -m pip list
Package Version Editable project location
------- ------- -------------------------
pip     24.0
xyz     0.1.0   /home/am/repos/xyz

... Sure enough, according to pip, my package xyz is there, has a version, and has an editable project location that resolved properly. Just to really drive the point home, here is the file tree for the path that pip manages to resolve taken from output of that command:

❯ tree /home/am/repos/xyz
/home/am/repos/xyz
├── pyproject.toml
├── README.md
├── requirements-dev.lock
├── requirements.lock
└── src
    └── xyz
        ├── __init__.py
        └── __pycache__
            └── __init__.cpython-312.pyc

3 directories, 6 files

[dsp]

Thanks @dsp, these are good tips.

When I cat the _xyz.path file, it's empty:

❯ cat .venv/lib/python3.12/site-packages/_xyz.pth

Interesting, on my computer with most recent rye and uv, it is not empty.

$ cat .venv/lib/python3.12/site-packages/_rye_test.pth 
/Users/dsp/src/rye-test/src

❯ rye run python3
Python 3.12.1 (main, Jan  8 2024, 05:57:25) [Clang 17.0.6 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pprint import pprint
>>> import sys
>>> pprint(sys.path)
['',
 '/home/am/.rye/py/[email protected]/install/lib/python312.zip',
 '/home/am/.rye/py/[email protected]/install/lib/python3.12',
 '/home/am/.rye/py/[email protected]/install/lib/python3.12/lib-dynload',
 '/home/am/repos/xyz/.venv/lib/python3.12/site-packages']

Because my .pth is not empty, I do have '/Users/dsp/src/rye-test/src' in my `sys.path.

I am honestly a bit at a loss here. One thing to try is to disable behavior.use-uv and see if clearing .venv + rye sync will do better. I am curious if this is a uv issue or something within rye. Since rye is actually a fairly simple wrapper, I tend to think it might be an issue with uv.

[mitsuhiko]

It sounds like uv is doing something wrong, but it might be that we are doing something wrong to uv to cause this. I definitely cannot reproduce this on my machine though. :(

[amontalenti]

Alright, at least we're getting warmer. First, I tried revising rye config to no longer use uv via config.toml. I then blew away the venv and remade it. That didn't change anything, even though I could see that pip-tools was used this time around:

❯ rm -Rf .venv
❯ rye sync
Initializing new virtualenv in /home/am/repos/xyz/.venv
Python version: [email protected]
Generating production lockfile: /home/am/repos/xyz/requirements.lock
Generating dev lockfile: /home/am/repos/xyz/requirements-dev.lock
Installing dependencies
Looking in indexes: https://pypi.org/simple/
Obtaining file:///. (from -r /tmp/tmp4zrathkf (line 1))
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: xyz
  Building editable for xyz (pyproject.toml) ... done
  Created wheel for xyz: filename=xyz-0.1.1-py3-none-any.whl size=953 sha256=8a9e5e70833aaad45862da7b43d2c16bf026d63fea18aab0213dec0ec3a63664
  Stored in directory: /tmp/pip-ephem-wheel-cache-095n6m3r/wheels/8b/19/c8/73a63a20645e0f1ed9aae9dd5d459f0f7ad2332bb27cba6c0f
Successfully built xyz
Installing collected packages: xyz
Successfully installed xyz-0.1.1
Done!
❯ rye run python3
Python 3.12.1 (main, Jan  8 2024, 05:57:25) [Clang 17.0.6 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import xyz
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'xyz'
>>>

OK, so uv doesn't seem to be the specific cause. Then, I wanted to confirm @dsp's idea that the .pth file being empty is responsible. Indeed, it is.

❯ ls
pyproject.toml  README.md  requirements-dev.lock  requirements.lock  src
❯ echo "$PWD/src" >.venv/lib/python3.12/site-packages/_xyz.pth
❯ rye run python3
Python 3.12.1 (main, Jan  8 2024, 05:57:25) [Clang 17.0.6 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import xyz
>>> xyz.hello
<function hello at 0x7f6be23ab560>
>>> xyz.hello()
'Hello from xyz!'
>>> 

So, not sure why neither pip-tools nor uv is putting a full path in that _xyz.pth file, but that's why this isn't working as expected.

[amontalenti]

Also just wanted to share all my version info that applies to my last couple comments:

❯ ~/.rye/pip-tools/[email protected]/bin/pip-sync --version
pip-sync, version 7.3.0
❯ ~/.rye/pip-tools/[email protected]/bin/pip-compile --version
pip-compile, version 7.3.0
❯ ~/.rye/pip-tools/[email protected]/bin/pip --version
pip 23.3.2 from /home/am/.rye/pip-tools/[email protected]/lib/python3.12/site-packages/pip (python 3.12)
❯ which rye
/home/am/.rye/shims/rye
❯ rye --version
rye 0.27.0
commit: 0.27.0 (43ee4fce0 2024-02-26)
platform: linux (x86_64)
self-python: [email protected]
symlink support: true
uv enabled: false

[dsp]

@amontalenti your UV sohuld come from ~/.rye/self/bin/uv or ~/.rye/uv/$VERSION/uv. That's what rye will call. It won't call the uv in $PATH.

[amontalenti]

@dsp Ah, good point:

❯ ~/.rye/self/bin/uv --version
uv 0.1.9

(Edited the original comment to remove uv version so there isn't any confusion.)

[jacobb]

@amontalenti I ran into something like this a bit ago, and it ended up being due to:

https://github.com/pypa/hatch/issues/1069

Personally, adding a .gitignore fixed it. Could you paste the contents of your project's root?

[amontalenti]

@jacobb VERY good find! That was it.

Here's why I was affected. I run one of those setups where my $HOME is a git repo with a $HOME/.gitignore containing just the line *. This is so that if I have configuration files (especially dotfiles) in my $HOME, I can version them by using git add -f as described in github.com/amontalenti/home.

Since I was just playing around with these rye projects, I didn't add them to version control or git. As a result, as you suggested, that bug in pypa/hatch was scanning for .gitignore above my project directory (which was ~/repos/xyz) and finding the one in my $HOME.

By doing git init . in the ~/repos/xyz directory, followed by making a no-op .gitignore file there with just a dummy ignored file file.tmp, suddenly rye sync did the right thing and created the _xyz.pth file appropriately, and thus import xyz worked. Mystery solved!

[amontalenti]

I was also able to confirm exactly your theory with this strace command:

strace -f -e 'trace=open,openat' rye sync 2>&1 | grep '.gitignore'

When there is .gitignore present in the project root, it opens that file. But when there is no gitignore present in the project root (when there is no $HOME/repos/xyz/.gitignore), it instead opens $HOME/.gitignore on my machine.

[mitsuhiko]

Yeah this unfortunately is a bug in hatch. I'm still considering whether hatch is a good default build tool choice, and if maybe setuptools would be a better default for now. For now I'm going to close this.