Not importing local files in a Lambda?

[Carmageddon]

trafficstars

Describe the Bug

I have stumbled into the problem that if I have a file called test_tenant_service.py file, and in it: from tenant_service import TenantService which would give error:

Could not find import of d/lambdas/tenants/test_tenant_service.py, looked in these locations (from config in /home/user/src/a/b/c/d/pyproject.toml): Search path (from config file): ["/home/user/src/a/b/c/d/database_lambda_layer"] Import root (inferred from project layout): "/home/user/src/a/b/c/d" Site package path (queried from interpreter at /home/user/src/a/b/c/d/.venv/bin/python): ["/home/user/.pyenv/versions/3.10.16/lib/python3.10", "/home/user/.pyenv/versions/3.10.16/lib/python3.10/lib-dynload", "/home/user/src/a/b/c/d/.venv/lib/python3.10/site-packages"]

Right there next to the file, there is a tenant_service.py file.. but its not detected. I made config settings to point to the common lambda layers that we use, as that makes sense, but adding every lambda to the config makes no sense..

Sandbox Link

No response

(Only applicable for extension issues) IDE Information

VS Code extension

[stroxler]

I think this may be working as intended, and we may need to understand the search path better.

I'm not very familiar with Lambda, but as a general rule if your project is rooted at some path /c/d, and you have modules at d/lambdas/tenants/tenant_service.py and d/lambdas/tenants/test_tenant_service.py, then the module name for the tenant_service.py code will be lambdas.tenants.tenant_service

My guess is something about your runtime is actually putting lambdas/tenants on the python path and we're not understanding this.

For what it's worth, sometimes it's hard to mimic all of the possible runtime paths of a python codebase in one configuration, and it's possible that this is one of those. When that happens, often a good workaround is to rely on relative imports, e.g. use from .tenant_service import TenantService. I've used this extensively when a codebase might live in more than one place depending on the context where it is used; the relative imports will always work as long as all the files live together.

[Carmageddon]

Basically, c is the repo root, with c/d, where code related stuff actually begins, but getting rid of d requires refactoring.

Anyway, with Lambdas: each lambda is kind of its own project, when deployed its only deployed from its own folder, so c/d/lambdas/tenants becomes its own root (+ it's lambda layers but I've handled that as I said).

I'll check again with . before imports, but IIRC, we settled on that because with the dot, it was causing import issues either during unit tests, or after deploying I dont remember.

However, with the above setup, pytest works from the /c/d/ directory - it can detect every test file in every lambda, and run it successfully - why Pyrefly cant handle it the same way pytest does?

Here is relevant pytest.ini config:

[pytest]
testpaths =
  lambdas
  tests/e2e
python_files = test_*.py
python_classes = Test*
python_functions = test_*

And coverage.rc:

[run]
branch = true

omit =
    app.py
    */__init__.py
    */alembic_migrations/*
    */config.py
    */app/*
    */tests/e2e/*
    */database_lambda_layer/*

Both live under /c/d/ directory.

So, if pytest can run this test with its import as they are, I think it is fair to expect Pyrefly to do the same, or at least be configurable to do the same.

[stroxler]

Makes sense, thanks for the context!

I think Pyrefly probably won't be able to do this out of the box (I suspect neither can pytest, which is why the config is needed).

But we probably do want to have a a way to set up a project of sub-projects; in this case the big project is top-level but every nested directory inside lambdas appears to be a sub-project rooted at that nested directory.

This might take us some time to figure out, it may require some careful thought about how to adjust our existing configuration.

[connernilsen]

Hey @Carmageddon, sorry, but I was out for part of last week, and I'm still catching up with stuff from this week. I'll get around to looking at this tomorrow or early next week. Just wanted to let you know that it's still in my queue!

[github-actions[bot]]

This issue has been automatically marked as stale because it is assigned, but has not had recent activity for more than 2 weeks.

If you are still working on this issue, please add a comment to keep it active. Otherwise, please unassign yourself and allow someone else to take over.

Thank you for your contributions!

[connernilsen]

Hey @Carmageddon, I still have plans to look at this, but I've recently been busy trying to catch up on a few things. I'll hopefully get around to it in the next week or two.

[connernilsen]

@Carmageddon, are you using a type checker currently that supports this mode of setup? I'm starting to get more time to research this, and I'm trying to see what else (other than Pytest) is currently available that supports this.

[connernilsen]

I'm going to remove myself from this for now, since I'm not going to be able to get to it in the near future. Can help plan it out/reevaluate if this should be a higher priority with more info and if others seem interested