`poetry build` follows symlinks for files but not for directories

[droserasprout]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: MacOS Catalina, gitlab/gitlab-runner:latest
• Poetry version: 1.0.3
• Link of a Gist with the contents of your pyproject.toml file:

Issue

We are changing package structure of our project and want to keep some compatibility aliases via symlinks to make migration easier. Project structure is the following (package_c contains symlinks only):

package_a:
    module_aa
    module_ab
package_b:
    module_ba
    module_bb
package_c:
    package_a  # --> ../package_a
    module_ba  # --> ../package_b/module_ba
    module_bb  # --> ../package_b/module_ba

pyproject.toml contains following lines:

packages = [
    { include = "package_a", from = "src" },
    { include = "package_b", from = "src" },
    { include = "package_c", from = "src" }
]

Resulting wheel contains copies of module_ba and module_bb at package_c path (which is fine) but package_c/package_a path is ignored completely.

Current behavior: file symlinks are silently followed, targets are copied; directory symlinks are silently ignored

Expected behavior: One of these:

1. All symlinks inside project directory are followed and targets are copied to wheel
2. Symlinks are ignored, warning printed during build
3. Symlinks are prohibited, attempt to build project containing symlinks fails

I know symlinks in wheels have no official support (https://github.com/pypa/pip/issues/3500) but it would be nice if poetry will be at least more verbose when handling them.

[mands]

Not sure if part of this ticket or should be a new one - however as mentioned above poetry follows file symlinks for wheels, however for sdists stores the link directly, breaking the link and causing divergent behaviour when building.

In both cases it ignores directory symlinks.

[martinxsliu]

I have a similar problem, also not sure if I should make a new issue for or not.

My use case is that I'm trying to integrate Poetry with the Bazel build tool. For context, Bazel creates sandboxes for each build and within this sandbox it places a symlink for every file needed by the build. This means that within the context of a Bazel build, everything including all the source files, pypyroject.toml, and poetry.lock will be symlinks to the actual files.

Trying to build either wheel or sdist in such a situation is broken right now. The first problem I run into is that path variables for source files are resolved to their targets, while the path variable for the project root is not. This inconsistency leads to an error stating something to the effect of files are not relative to the root.

The second problem, specifically for sdist, is what @mands mentions. Links are not followed when creating a tarball and result in broken archives.

Going by https://github.com/pypa/pip/issues/5919, it doesn't appear like there's consensus on how pip should handle symlinks in archives, and the current behaviour feels like the unintended consequence of how Python's tar and zip packages behave. As such I feel like Poetry has an opportunity to take a more consistent and disciplined approach without the burden of legacy packages, at least until there is an unifying PEP about how symlinks ought to be handled.

I propose that we follow symlinks all the time for both wheel and sdist builds. Meaning that the file and directory targets of symlinks will be copied into the archives in the place of the symlink. Doing so will address all the issues raised here.

Any thoughts?

[brechtm]

I propose that we follow symlinks all the time for both wheel and sdist builds. Meaning that the file and directory targets of symlinks will be copied into the archives in the place of the symlink. Doing so will address all the issues raised here.

Are there any arguments against doing this? If not, this is an easy fix and I should be able to provide a PR.

[johanssone]

Will just add to this that I'm hitting the same issue. I'd love to help with a PR, but a bit lost.

Would https://github.com/python-poetry/poetry-core/blob/master/src/poetry/core/masonry/builders/builder.py#L173 be a starting point?

[pprados]

some news ?

[ttamg]

I am also having this issue. I have a symlinked directory in my src build folder. And I get the xxx does not contain any element error as it seems poetry build is not following the directory symlink.

Any update on this issue or do we need to use a different build system>

[Secrus]

Adding this would be nice for sure, although, judging by the age of the ticket, I would say there was no one determined enough to implement this.

[ttamg]

I've found a workaround which works for my use case here which is a multi-repo. I am posting here in case useful to others. My use case is to include packages from outside the pyproject.toml route when building. So this is an example project structure

package_a/
   pyproject.toml

package_b/
   pyproject.toml

src/
   my_code
   more_code

The symlink idea was to link in the relevant code from my src folder into a subdirectory in the package_a and package_b.

Instead I am using @davidvujic Poetry multiproject Plugin which allows me instead to keep the above structure without any symlinks, and link in the source code I need in the pyproject.toml files using:

packages = [ { include = "my_code", from = "../src"} ]

Normally this would fail with poetry build due to the import ../ from outside the pyproject.toml root, but with this plugin you can instead use the additional poetry build-project command and it works well for my use case.

The plugin is an easy install on poetry using poetry self add poetry-multiproject-plugin