Provide nix flake

[yajo]

We could also provide Copier as a nix flake, making it 100% reproducible under any environment.

Originally posted by @Yajo in https://github.com/copier-org/copier/issues/581#issuecomment-1053521572

[jmgilman]

Just FYI, the mkdocstrings downstream package has a circular dependency that Nix does not inherently support. See this issue for more details. I believe the circular dependency can be broken by patching the legacy package out, but I haven't tested it yet. Otherwise, you'll need to target an older version where the author didn't have circular dependencies.

Also, see this PR for a start to getting Copier added to NixPkgs. It's very large, which I think is contributing to it taking so long to merge, so I've started adding PRs for individual dependencies to try and speed up the process.

[pawamoy]

The circular dependency is only temporary, as explained in the linked issue (just re-stating it here for better visibility). mkdocstrings version 0.19 should be free of circular dependencies :slightly_smiling_face: Some work to do before getting there, but it should not take months.

[yajo]

You shouldn't need that dependency except for making the docs. You should be able theoretically to build and use copier without it. Have you tried that?

[jmgilman]

Yeah, I realized this the other day when I was looking at the pyproject.toml again and noticed the optional attribute. Poetry is confusing in that they make you add the dependency to the runtime section if you want to add it to an extras section (or at least that's what I've come to understand).

I'm working on slowly adding the dependencies to nixpkgs, my first one just went through today. Once I get to critical mass I'll add a copier package and update this thread.

Note that adding dependencies to nixpkgs isn't strictly necessary, I'm sure poetry2nix could accomplish the same thing in a flake. However, I use Nix to manage my development environment, so it's nice to be able to have it installed without needing a dedicated flake file.

[yajo]

Cool! well, if Copier gets maintained in nixpkgs directly, maybe a flake here won't make much sense, i don't know... 🤔

[yajo]

FTR yesterday I was trying to do a fix, and it seems like poetry itself can't resolve dependencies.

Just check it out with poetry update and see there are some failures. Also I can't serve with mkdocs, and both problems seem related:

(.venv) yajo@yajolap ~/m/copier (update-delete)> poetry update
Updating dependencies
Resolving dependencies... (10.5s)<debug>PackageInfo:</debug> Invalid constraint (importlib-metadata (>='4.4') ; python_version < "3.10") found in markdown-3.3.5 dependencies, skipping
Resolving dependencies... (14.2s)

Writing lock file

No dependencies to install or update
(.venv) yajo@yajolap ~/m/copier (update-delete)> poe docs
Poe => mkdocs serve
Traceback (most recent call last):
  File "/var/home/yajo/mydevel/copier/.venv/bin/mkdocs", line 5, in <module>
    from mkdocs.__main__ import cli
  File "/var/home/yajo/mydevel/copier/.venv/lib/python3.10/site-packages/mkdocs/__main__.py", line 13, in <module>
    from mkdocs.commands import build, gh_deploy, new, serve
  File "/var/home/yajo/mydevel/copier/.venv/lib/python3.10/site-packages/mkdocs/commands/build.py", line 12, in <module>
    from mkdocs.structure.nav import get_navigation
  File "/var/home/yajo/mydevel/copier/.venv/lib/python3.10/site-packages/mkdocs/structure/nav.py", line 4, in <module>
    from mkdocs.structure.pages import Page
  File "/var/home/yajo/mydevel/copier/.venv/lib/python3.10/site-packages/mkdocs/structure/pages.py", line 6, in <module>
    import markdown
  File "/var/home/yajo/mydevel/copier/.venv/lib/python3.10/site-packages/markdown/__init__.py", line 29, in <module>
    from .core import Markdown, markdown, markdownFromFile  # noqa: E402
  File "/var/home/yajo/mydevel/copier/.venv/lib/python3.10/site-packages/markdown/core.py", line 26, in <module>
    from . import util
  File "/var/home/yajo/mydevel/copier/.venv/lib/python3.10/site-packages/markdown/util.py", line 86, in <module>
    INSTALLED_EXTENSIONS = metadata.entry_points().get('markdown.extensions', ())
  File "/usr/lib64/python3.10/importlib/metadata/__init__.py", line 980, in entry_points
    return SelectableGroups.load(eps).select(**params)
  File "/usr/lib64/python3.10/importlib/metadata/__init__.py", line 429, in load
    ordered = sorted(eps, key=by_group)
  File "/usr/lib64/python3.10/importlib/metadata/__init__.py", line 977, in <genexpr>
    eps = itertools.chain.from_iterable(
  File "/usr/lib64/python3.10/importlib/metadata/_itertools.py", line 16, in unique_everseen
    k = key(element)
AttributeError: 'PathDistribution' object has no attribute '_normalized_name'

Maybe poetry2nix is not so buggy and there's a real problem here?

[jmgilman]

The markdown problem looks like it was discussed here and solved in v3.3.6. I think the mkdocs error is related to importlib_metadata.

[jmgilman]

Your mkdocs error is due to running Python 3.10 which breaks with lower versions of importlib-metadata. I was able to resolve the dependency issue by updating pyproject.toml as follows:

[tool.poetry.dependencies]
python = ">=3.9,<4.0"
# ...
importlib-metadata = "^4.11.3"
# ...

[yajo]

It seems the upstream issue is https://github.com/Python-Markdown/markdown/issues/1227. In any case, I guess that if the flake uses python 3.9, it will have no problem.

[rszamszur]

@yajo I do not know flakes yet, but if you'd like I can contribute with Nix expressions for building Poetry package + shell with an editable package for development. Also to have deterministically installed poetry2nix, I can pin the version in nixkpgs overlay (I think it's better that way since poetry2nix is updated through nixpkgs, and also its easier to add overrides "globally"). Lastly, whenever possible I'd pin the default nixpkgs to concrete commit sha. I guess those things should provide full reproducibility. Let me know what you think.

Have a good one!

[yajo]

Hello! Well, I'm totally into flakes. For me, it makes more sense to do it the opposite way: using https://github.com/edolstra/flake-compat to allow downgrading to stable nix when flakes support can't be used.

Besides I have made https://gitlab.com/moduon/precommix which is itself a copier template that will help creating the flake and replacing the bad parts of pre-commit. FWIW you can download an (outdated) copier package from that repo (for now).

So my intention is to use those tools. I'm open to contributions, but I would really like them to be done this way, because it's gonna be a way to also stress those tools and see if how they fit in the outside world.

In #696 you can see an initial attempt (which failed, BTW, and is stale now, but I will resurrect it when I have time). You can draw inspiration from there. Also you'll probably need something like https://github.com/nix-community/poetry2nix/issues/768#issuecomment-1290721575 (which I think should get to upstream poetry2nix).

[rszamszur]

Roger that! Well, it is about time I learn nix flakes so win-win :D. When I'll be back from holiday and this issue still will be open I can give it a try.

[rszamszur]

Hey @yajo

Is this ticket still open for contribution? I saw you're still working on #696 and just want to double-check. If open, then are you interested in a workflow for testing nix flake as well (nixpkgs-fmt and nix build)?

[yajo]

I definitely want testing, but it's something I'll be doing there too. I think it's better to wait until #696 is finished, so further improvements can be more scoped.

[rszamszur]

Sure, let me know if you need any help.

In the meantime few resources you might find useful:

• Cachix - Nix binary cache. They offer a free 5 GB plan for open-source projects, and it's easy to set up.
• https://github.com/cachix/install-nix-action
• https://github.com/cachix/cachix-action
• poetry2nix flake template
• Example Nix workflow - far from ideal, but should help you get started.

Lastly, you can ping me once the flake is ready. I can test it on aarch64-linux and aarch64-darwin platforms.

Have a good one!

[yajo]

I ended up using Cachix and Devenv, instead of Precommix. Thanks!