bazelbuild
docs(examples): Add example of using a `src` dir and separate `tests` dir with gazelle
Add an example of using a src dir and separate tests dir and
having gazelle generate targets correctly so that tests can be run.
Fixes #1775.
I can't seem to figure out why builds are failing. Any tips?
If I run bazel test //... from the new example directory, things run just fine.
$ cd examples/bzlmod_python_src_dir_with_separate_tests_dir/
$ bazel test //...
INFO: Analyzed 10 targets (4 packages loaded, 24 targets configured).
INFO: Found 7 targets and 3 test targets...
INFO: Elapsed time: 20.303s, Critical Path: 18.58s
INFO: 13 processes: 7 internal, 2 local, 4 processwrapper-sandbox.
INFO: Build completed successfully, 13 total actions
//:gazelle_python_manifest.test PASSED in 0.2s
//:requirements_test PASSED in 16.2s
//tests:test_my_python_module PASSED in 0.6s
Executed 3 out of 3 tests: 3 tests pass.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.
If I run from the git repo root, I get the same failures seen in CI (unknown repo 'pypi').
$ cd -
/c/dev/rules_python
$ bazel test //...
ERROR: Skipping '//...': error loading package under directory '': error loading package 'examples/bzlmod_python_src_dir_with_separate_tests_dir': Unable to find package for @@[unknown repo 'pypi' requested from @@]//:requirements.bzl: The repository '@@[unknown repo 'pypi' requested from @@]' could not be resolved: No repository visible as '@pypi' from main repository.
ERROR: error loading package under directory '': error loading package 'examples/bzlmod_python_src_dir_with_separate_tests_dir': Unable to find package for @@[unknown repo 'pypi' requested from @@]//:requirements.bzl: The repository '@@[unknown repo 'pypi' requested from @@]' could not be resolved: No repository visible as '@pypi' from main repository.
INFO: Elapsed time: 25.549s
INFO: 0 processes.
ERROR: Build did NOT complete successfully
ERROR: Couldn't start the build. Unable to run tests
From what I can tell, the only differences between this example and bzlmod or bzlmod_build_file_generation is that this example:
- Doesn't have a WORKSPACE file
- Uses
pypifor the hub name instead ofpip.
But I tried making both of those edits locally and still got the same error.
Note: I'm ignoring the MacOS and Windows CI failures for now.
Please run the pre-commit hooks that will update .bazelignore, which should fix the errors seen in the tests running from the root.
Edit: or the .bazelrc, can't remember which one in this case.
Thanks. Turns out it's both .bazelrc and .bazelignore.
I also noticed that the pre-commit hook update-deleted-packages doesn't appear to work correctly, so I've opened #1858 with details.
I've been thinking about this PR and how do we organise it together with other gazelle docs that we have. We are running short on CI executors so just adding this example into the CI is not so straight forward. For running bazel-in-bazel integration tests we use https://github.com/bazel-contrib/rules_bazel_integration_test. However, this makes our tests slow to execute as they become yet another thing that needs to be executed when we do bazel test //....
What is more, if we merge this PR, does it trump the other example on build_file_generation? What is the expected example people are to follow? Given that the Python community is recommending this structure, does it mean that we should follow their recommendations as well? bazel has different limitations and benefits and I am wondering about the tradeoffs here and in other places.
Given that this example may be more complex than the existing gazelle example I would be +1 on replacing the existing example with this or merging them in some way.
Sorry that this is dragging so long, but I am a bit lost with how we teach our users to use gazelle and it seems that more and more people are starting to use it, so your PR is a great addition to the knowledge base, but I am concerned that if we are not running CI on it, it will become stale and will cause new issues about "stale example".
Am I overthinking it here? What are your thoughts? We can have a call to chat this because I do think that having an async messaging is slowing us down. Shall we get in touch on the bazel slack?
I don't have a slack account, sorry. I could do Google Meet if that works for you.
No, you're not overthinking this at all - you have great concerns and know a bunch more about the project than I do. For example, I didn't know about bazel-in-bazel (though I should have suspected, haha) or about how there are a limited number of CI executors.
does it trump the other example? What is the expected example people are to follow?
I would say that there should be a single "main" example to follow, ideally with a limited feature set consisting of the most common use cases (how many projects need whl_mods from the start?)
IMO the feature set for the main example should be:
- python toolchain (hermetic python)
- pip.parse, obviously
- gazelle python manifest
- gazelle and some common directives
We don't need the requirement locking (compile_pip_requirements). At least IME, large projects already have that via tools like poetry or uv.
should Bazel follow the python community recommendation for project structure?
I'd say that depends on the goals of rules_python.
Do people want rules_python to be used in more and more open-source projects (eg: get numpy to switch from meson to bazel; get flask to use it; have it be the default for greenfield projects)? If so, then the "main" example should follow community recommendation for project structure, and modifying as necessary to address the Basel-specific directory structure needs.
Or is rules_python targeting more closed-source systems, like Google and Uber and Dropbox, etc.? If so, then the "main" example should probably follow the a structure that more closely reflects what is done in corporate environs (which would be harder to figure out).
I would be +1 on replacing the existing example with this or merging them in some way
Same. When we make a decision I can work on that.
I am concerned that if we are not running CI on it, it will become stale
+100 on that. Incorrect docs can be worse than no docs, haha.
Decision from our video chat: integrate this example into the existing examples/bzlmod_build_file_generation example with a dir structure somewhat like:
./examples/bzlmod_build_file_generation
+ pattern1/
+ (the current example)
+ pattern2/
+ the example from this PR
+ README.md # includes descriptions of the patterns