Support for Editable Dependencies

[RobertRosca]

As far as I could tell, there's currently no way to specify that a dependency should be installed in editable mode, which would be a useful feature for monorepo-style projects.

A practical example of this is from Jupyverse, which has a monorepo structure with multiple packages in a ./plugins/ directory. Here are the relevant sections from pyproject.toml for Jupyverse:

[build-system]
requires = [ "hatchling",]
build-backend = "hatchling.build"

[project]
dependencies = [
    "fastapi>=0.82.0",
    "fps>=0.0.19",
    "fps-uvicorn>=0.0.19",
    "fps-auth-base>=0.0.42",
    "fps-contents>=0.0.42",
    "fps-kernels>=0.0.42",
    "fps-terminals>=0.0.42",
    "fps-nbconvert>=0.0.42",
    "fps-yjs>=0.0.42"
]

[project.optional-dependencies]
jupyterlab = [ "fps-jupyterlab >=0.0.42",]
retrolab = [ "fps-retrolab >=0.0.42",]
auth = [ "fps-auth >=0.0.42",]
auth-fief = [ "fps-auth-fief >=0.0.42",]
noauth = ["fps-noauth >=0.0.42"]
test = [ "mypy", "types-setuptools", "pytest", "pytest-asyncio", "pytest-env", "requests", "websockets", "ipykernel",]
docs = [ "mkdocs", "mkdocs-material",]

[tool.hatch.envs.dev]
pre-install-commands = [
  "pip install -e ./plugins/auth_base",
  "pip install -e ./plugins/contents",
  "pip install -e ./plugins/frontend",
  "pip install -e ./plugins/kernels",
  "pip install -e ./plugins/lab",
  "pip install -e ./plugins/nbconvert",
  "pip install -e ./plugins/terminals",
  "pip install -e ./plugins/yjs",
]
dependencies = ["fastapi>=0.82.0"]
features = ["test"]

[tool.hatch.envs.dev.overrides]
matrix.auth.post-install-commands = [
  { value = "pip install -e ./plugins/noauth", if = ["noauth"] },
  { value = "pip install -e ./plugins/auth", if = ["auth"] },
  { value = "pip install -e ./plugins/auth_fief", if = ["auth_fief"] },
]
matrix.frontend.post-install-commands = [
  { value = "pip install -e ./plugins/jupyterlab", if = ["jupyterlab"]},
  { value = "pip install -e ./plugins/retrolab", if = ["retrolab"]},
]

[[tool.hatch.envs.dev.matrix]]
frontend = ["jupyterlab", "retrolab"]
auth = ["noauth", "auth", "auth_fief"]

For development you'd want to install the plugins in editable mode from the ./plugins/ directory, which is why [tool.hatch.envs.dev] defines a pre-install-command which runs pip install -e ./plugins/PLUGIN_NAME on core plugins, and then tool.hatch.envs.dev.overrides defines matrix post-install commands for specific additional packages.

As mentioned in this discussion post https://github.com/pypa/hatch/discussions/516, there are a few ways I can think of to allow for this:

1. Extend dev-mode-dirs to work with environment installation as well as builds.
2. Add another dependency category, something like editable-dependencies, which would always perform a pip install -e ... on the provided dependency.
3. Additional syntax for the dependency specification, maybe -e proj @ URI, which when found installs the dependency in editable mode.

Out of these three, I think 2 and 3 make the most sense as they are more explicit over what's happening.

editable-dependencies Category

The additional category approach would add sections for editable-dependencies and extra-editable-dependencies to the configuration file. This would look like:

[tool.hatch.envs.dev]
editable-dependencies = [
  "fps-auth-base @ {root:uri}/plugins/auth_base",
  "fps-contents @ {root:uri}/plugins/contents",
  ...
]

[tool.hatch.envs.dev.overrides]
matrix.auth.extra-editable-dependencies = [
  { value = "fps-noauth @ {root:uri}/plugins/noauth", if = ["noauth"] },
  ...
]

Support -e Flag for Local Dependencies

The additional syntax approach would use the existing sections, but support the -e flag:

[tool.hatch.envs.dev]
dependencies = [
  "-e fps-auth-base @ {root:uri}/plugins/auth_base",
  "-e fps-contents @ {root:uri}/plugins/contents",
  ...
]

[tool.hatch.envs.dev.overrides]
matrix.auth.extra-dependencies = [
  { value = "-e fps-noauth @ {root:uri}/plugins/noauth", if = ["noauth"] },
  ...
]

In a way this is odd as it isn't really specified in any PEPs, however PDM uses this syntax for editable dependencies already, so it isn't that odd. In PDM the above dependencies would look like:

[tool.pdm.dev-dependencies]
dev = [
    "-e file:///${PROJECT_ROOT}/plugins/auth_base#egg=fps-auth-base",
    "-e file:///${PROJECT_ROOT}/plugins/contents#egg=fps-contents",
]

Next Steps?

I'm happy to implement either of these two options, but not sure which one to go with.

The -e flag support is pretty nice and avoids requiring a new section, however having that flag in a requirement it isn't officially defined in any PEP, but on the other hand it is already used in PDM.

Additional categories are much clearer but make the files a bit more verbose.

Personally I prefer the -e flag since it is already used in PDM which is quite popular.

[rademacher-p]

Hi, all 👋 I'm attempting to get around the lack of editable support myself and have a slightly different issue. Here's a pyproject.toml for main package foo trying to install local dependency bar:

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
dependencies = [
  "bar @ {root:uri}/lib/bar",
]

[tool.hatch.metadata]
allow-direct-references = true

[tool.hatch.build]
ignore-vcs = true
packages = ["src/foo"]

[tool.hatch.envs.default]
post-install-commands = ["pip install -e lib/bar"]

I like having bar in project.dependencies so that it will get installed (not editable, though 😢) using a normal pip install -e <path-to-foo>. For hatch users, I use post-install-commands to reinstall bar in editable mode. However, when hatch synchronizes the dependencies, it reinstalls bar yet again, reverting to non-editable.

Any ideas how I can get hatch to accept the editable install and not observe an out-of-date dependency? As a temporary measure, can the dependency sync be turned off and controlled manually?

Thanks for your help 🙏

[janhurst]

i have exactly @rademacher-p's use case

[tmke8]

I wonder whether local dependencies for hatch environments should always be installed in editable mode? Is there a scenario where you wouldn't want that? Hmm, I guess for a build environment you might not want that...

[janhurst]

i do have a use case where i have a submodule in a monorepo that is a patched upstream package... i don't need it to be an editable install. (this is a really horrible workaround mind you)

[jrocketdev]

Any movement on this idea that I'm missing? I saw @ofek has some ideas on the #589 but it's been closed.

I'd be happy to help if we can agree on an approach but don't want to waste too much time going in a direction that doesn't have support. The ability to have editable local installs in the dependencies setting would be great for a few projects I'm on. Specifically those following a more mono-repo approach that has inter-repository dependencies.

I currently workaround the issue by using post-install-commands or pre-install-commands and pip install -e ....

Example:

pre-install-commands = [
  "pip install -e ../{local-package-here}/"
]
dependencies = [
  ...
  "local-package-here",  # This gets skipped because of the pre-install command
]

However, this approach isn't always robust because additions and modifications to the pre/post install commands don't register as an environment change (for good reasons) so any modifications require a removal of the current env and then a recreation from scratch. This is not super user friendly and local editable installs in the dependencies would make this much cleaner.

[levchik]

Hi all, first of all, thanks for the wonderful work on hatch.

I also use pkg[extra] @ {root:uri}/../pkg but for me pre-install-commands or post-install-commands do not even work correctly as a workaround.

After I did env remove and env create, my package still uses some old (cached somewhere maybe?) version of the dependency pkg.

The only way I managed to make it work was to go into hatch shell and run pip install -e ../app-pkg[extra]. After that it is ok, but any change in pkg requires doing this over and over again.

Environment

Hatch, version 1.7.0

Python 3.11.3

Mac OS 13.4

[alexclaydon]

We're all in on Hatch and this issue is the only showstopper for us as we have a number of monorepos. The suggestions from @RobertRosca - in particular 2 and 3 - seem like elegant solutions. Are there any plans to carry this forward?

[potiuk]

It might not be a final solution - but I've just merged a huge PR for Airflow's monorepo where I implemented this very feature in a slightly workaround'y way, but working really nicely for us. Probably this solution is not for a feint of heart but since we have no direct support for different kinds of dependencies (editable/non-editable) in pyproject.toml, using custom build hooks, it might quite a good solution for a number of cases where you have complex monorepo.

The gist of it is in this hatch_build.py file in Airlfow repo: https://github.com/apache/airflow/blob/main/dev/hatch_build.py#L147

In short it works in this way:

a) airflow dependencies in pyproject.toml are generally the "editable" ones - NOT the final ones that land in the .whl file. There are subtle (and not so subtle) differences between the two sets.

b) our custom build hook checks if the build initialization is "standard" (i.e. non-editable) - and then we modify the depedencies dynamically based on a json specificaition of dependencies and few other rules.

This is the gist of the code:

# remove devel dependencies from optional dependencies for standard packages
self.metadata.core._optional_dependencies = {
    key: value
    for (key, value) in self.metadata.core.optional_dependencies.items()
    if not key.startswith("devel") and key not in ["doc", "doc-gen"]
}
# Replace editable dependencies with provider dependencies for provider packages
for dependency_id in DEPENDENCIES.keys():
    if DEPENDENCIES[dependency_id]["state"] != "ready":
        continue
    normalized_dependency_id = dependency_id.replace(".", "-")
    self.metadata.core._optional_dependencies[normalized_dependency_id] = [
        f"apache-airflow-providers-{normalized_dependency_id}"
    ]
# Inject preinstalled providers into the dependencies for standard packages
if self.metadata.core._dependencies:
    for provider in PREINSTALLED_PROVIDERS:
        self.metadata.core._dependencies.append(provider)
    for dependency in PREINSTALLED_NOT_READY_DEPS:
        self.metadata.core._dependencies.append(dependency)

That allows us to:

1. remove devel_* dependencies from the wheel
2. add some preinstalled dependencies that should only happen when we install airflow from .whl but not when installing it in --editable mode (that allows us to avoid installing deps that we have directly available in our monorepo from airflow sources
3. replace some devel "extra" sets of dependencies with target packages in the "other from monorepo" .whl packages - this way when installing deps in --editable mode, we only install dependencies of our "monorepo" packages but when installing from regular .whl we install target packages (which has those dependencies).

Happy to share more details if needed.

[tmke8]

If anyone is desperately looking for a Python package manager supporting monorepos, rye has support for this now via workspaces: https://rye-up.com/guide/workspaces/

Like hatch, rye follows the standards for pyproject.toml specified in PEP 621, so at least the [project] table should already be compatible.

[potiuk]

If anyone is desperately looking for a Python package manager supporting monorepos, rye has support for this now via workspaces: https://rye-up.com/guide/workspaces/

Like hatch, rye follows the standards for pyproject.toml specified in PEP 621, so at least the [project] table should already be compatible.

Nice. Seems like we are getting to the point where there is a lot of effort in the tooling around the packaging standards (think uv for example) where you can freely choose your tooling without being heavily dependent on it, and this is fantastic outcome of all the work done by Packaging team to develop and implement standards.

I will certainly have a look at that, it could be exactly what we need from the first glance - for Airlfow case at least.

Someone mentioned https://polylith.gitbook.io/polylith/ as another way of looking at monorepos, but I personally find it far too opinionated and far too much of your application depends on being a "polylith" application. It seems good if you are building an inernall application split in multiple loosely ependent pieces, but anyhow yoy have to architect your application about being a "ploylith" app.

The rye approach with workspaces seems to be very, very lightweight, where you merely make sure that you install separate, independently developed packages that can be "installed" and "worked on" together - without the heavy reliance on architecting your whole app in an opinionated way. Thanks for the pointer @tmke8

[x0ul]

Consider this one more vote in favor of this kind of feature, or perhaps workspaces.

I work on several python packages, call them AppA, AppB, etc., all of which depend on CommonLibs. Often when working on AppA I want to make some changes to CommonLibs locally and have those changes immediately available in AppA. These packages are all stored in separate repos and checked out into a common parent dir, my_workspace.

Right now the pyproject.toml file in AppA, etc. point to the server hosting the CommonLibs repo so they can be built by CI pipelines. If I'm developing locally it'd be really nice to have a way for the locally checked-out copy of CommonLibs override the remote CommonLibs. Right now I don't have a solution to achieving this functionality with hatch, but it seems like the feature suggested in this issue might help. Alternatively, the workspaces feature of rye mentioned above might work for me but I'm not too keen to switch build systems right away.