astral-sh
uv workspaces and namespace packages
TLDR: I'm trying to build multiple packages under the same namespace using
uvworkspaces, but I'm having trouble. I would greatly appreciate it if the documentation included instructions on how to handle namespace packages.
I've been tinkering with workspaces since they were released, but I have needed help to get what I wanted working. Here is my use case. I am building a core package called cable. I would also like to build a command-line interface for cable that should be an optional dependency of cable. If installed, it should be importable as cable.cli. I also want to build another "extension" called experiments.
Both the cli and experiments packages depend on the core cable code and I don't want the cable package to know about cli and experiments or their specific dependencies (i.e. typer, mlflow).
I want to use workspace to cleanly isolate the code and dependencies, but I have struggled to build the packages within my cable namespace.
Another question/thought is that I don't quite understand why all my workspace members must have their own versions. In my use case, the only version that matters is the cable version.
Am I misunderstanding how/when to use workspaces?
Your example is a great use case for workspaces. I've create am example repo for this: https://github.com/konstin/uv-workspace-example-cable. Feel free to ask more questions here!
Most information should be in the readme, copied below. I've chosen not to user namespace packages but isolated packages, in my experience namespace packages cause more trouble than they solve.
uv workspace example: cable
This project contains a workspace with four packages: cable, cable-experiments, cable-cli, and cable-core. cable is the main project that users install. It has extras cable[cli] and cable[experiments] that pull in cable-experiments and cable-cli respectively. cable-core implements shared utils.
Since cable is the main project, it lives in the workspace root (not all workspaces have a root, but for many project a clear root that (indirectly) depends on everything else is very convenient), while all other packages live in packages.
---> cable-cli -----------|
cable --| |--> cable-core
---> cable-experiments ---|
or with uv tree:
cable v0.1.0
├── cable-cli v0.1.0 (extra: cli)
│ └── cable-core v0.1.0
└── cable-experiments v0.1.0 (extra: experiments)
└── cable-core v0.1.0
When the user installs cable, they can decide if they want from a minimal installation (cable) to all features (cable[cli,experiments]).
The project offers an optional CLI. The script entrypoint lives in cable-cli, because there's a current limitation that scripts can't depend on extras, so cable-cli installs a cable <name> script; It's still installed through cable[cli].
You can create this workspace structure roughly with:
uv init --lib cable
cd cable/
mkdir packages
cd packages/
uv init --lib cable-core
uv init --lib cable-cli
uv init --lib cable-experiments
cd cable-cli/
uv add ../cable-core
cd ../cable-experiments/
uv add ../cable-core
cd ../..
uv add --optional cli packages/cable-cli/
uv add --optional experiments packages/cable-experiments/
uv sync --all-extras
This is great, and almost what I'm looking for as well. I have one question though: if I'd like to add a project to this structure that is, say, a website that depends on cable, how would I go about adding it? My use case is in the end to serve the website via a docker image that installs cable as a dependency. Would this also fit the use case of workspaces? I do not want to add the website as a dependency or an optional dependency, because it's just a client of cable, but would like to use a monorepo structure to keep it in sync with the development of cable.
In that case, you probably want either the website as root and cable as subfolder, or all packages in subfolders (e.g. a dedicated packages), while the root has no packages; The details depend on the details of your project workflows.
Thanks @konstin , I will probably have multiple websites / UIs that depend on the core package and possibly some add-ons. Would you recommend in that case to split the package to have a root library that depends on the core set of packages, then a set of packages in a subfolder packages, and another subfolder with a set of UIs / websites which are not packages?
I also have a question about the presence of the src folder. Is there an actual reason for it? After all if I have folder/packages/package_name/package_name why add another layer and do folder/packages/package_name/src/package_name/?
I'd start with a flat layout where all packages are directly in packages.
After all if I have folder/packages/package_name/package_name why add another layer and do folder/packages/package_name/src/package_name/?
When you're in folder/packages/package_name/, this ensures that import package_name will import from the venv and not from package_name, so you're always going through the editable instead of the import depending on where you're in the package root or not, see https://packaging.python.org/en/latest/discussions/src-layout-vs-flat-layout/.
in my experience namespace packages cause more trouble than they solve
Can you elaborate on, what problems they might cause, @konstin? If one were to deeply desire namespace'd packages, how would you implement them using uv and workspaces?
I am also curious about the problems from @konstin's experience?
Perhaps one of the issues could be the easy mistake of keeping __init__.py files in the root of the package, resulting in an invalid namespace package. This is deceptive because it appears to work fine when the packages are installed normally, but this is unintended behaviour, and it does not work for editable installs as explained here.
Namespace packages do have the limitation of not being able to have a root-level __init__.py file containing some logic, as described here, so in some cases, isolated packages could be preferred.
I set up a repo containing a namespace package example setup that seems to work within a uv workspace, loosely based on the cable example above. It does, however, require changing the core's __init__.py file to a core.py file.
https://github.com/ribeirompl-soldersmith/uv-namespace-package-workspace
Namespace packages can work by overriding each others files, so you get the file of the package that was installed last; One package can modify a directory also used by another package instead of isolation between them. This makes tooling around them harder, e.g. editable installs, caching, layered installation (since the namespace packages with the same "root package" need to be installed together) or figuring out which package is missing. My perspective is colored by being a tool author more than a library author, but i'd go for importlib.metadata.entry_points or something similar for plugins over interleaving mechanisms such as namespace packages.
@konstin There is a point from the original question that is not being addressed in your example which is that the packages should share the same namespace name (here cable).
Which would results in the following imports:
import cable.core
import cable.cli
import cable.experiment
As opposed to:
import cable_core
import cable_cli
import cable_experiment
Changing the folder structure to reflex that (e.g. cable/core instead of cable_core) results in an error with uv sync:
× Failed to build [...]
[...]
The most likely cause of this is that there is no directory that matches the name of your project (cable_core).
Implicit namespace names accross libs is a valid Python spec (PEP 420).
Do you confirm this is currently not possible with UV workspaces or am I missing something?
I'd have to see the complete new tree and command output, but did you check that the names in project.name as well as [tool.uv.sources] match your updated structure?
Well, you do need unique names under [tool.uv.sources], right? So it cannot reflex the following structure:
If we keep the same example, the tree would be:
├── packages
│ ├── cable-core
│ ├── pyproject.toml
│ └── src
│ └── cable
│ └── core
│ └── __init__.py
│ └── cable-assets
│ ├── pyproject.toml
│ └── src
│ └── cable
│ └── cli
│ └── __init__.py
├── pyproject.toml
Which is valid under across multiple libs based on PEP 420, if you were to install cable-core and cable-assets separately. They would share the same root namespace.
Adding the following to the packages's pyproject.toml does solve the build issue, which makes sense.
[tool.hatch.build.targets.wheel]
packages = ["src/cable"]
Which then allows to use a shared root namespace.
omg that was too hard! here's my example.
- flattest possible file structure
- all sub-package
__version__s are the same as 'core'. - sucks: had to spell out deps in the main pyproject.toml instead of in its respective sub-package pyproject.toml. else, the distributable (whl) will not count them as deps of the 'core' pkg (but deps of their respective ns pkgs)
- sucks: ...ditto for optional script def
toml with references would have helped keep things in their respective locations!
omg that was too hard! here's my example.
- flattest possible file structure
- all sub-package
__version__s are the same as 'core'.- sucks: had to spell out deps in the main pyproject.toml instead of in its respective sub-package pyproject.toml. else, the distributable (whl) will not count them as deps of the 'core' pkg (but deps of their respective ns pkgs)
- sucks: ...ditto for optional script def
toml with references would have helped keep things in their respective locations!
Hi @majidaldo, thanks for your example looks nice and was really helpful to me, the only thing that I've changed is versioning.
I'd prefer to use env instead path it every time
[tool.hatch.version]
source = "env"
variable = "VERSION" <- Your var here
https://hatch.pypa.io/1.13/plugins/version-source/env/#pyprojecttoml
I wasn't aware that uv normalizes dotted packages foo.bar to foo-bar.
For future reference: Here is my demo-example: https://github.com/libranet/demo-uv-workspace/tree/main
