[Docs] Lack of fully fledge example with actually working dev env and build setup

[aMOPel]

Describe the issue

What do I mean with actually working developer environment?

A shell that:

1. contains the python interpreter installed from nix wrapped with the pinned python packages. The problem is, that some python packages require correctly linked dynamic C libraries to work. Simply importing python and python packages from nixpkgs separately won't work. There is a work around with nix-ld, however as I understood it, that is why python.withPackages exists, which handles this wrapping for you. poetry2nix uses python.withPackages in mkPoetryEnv.

2. doesn't require the developer to send the developed package through the nix-store again, after every change. If the derivation of the developed package itself is put into the nix-shell, you have to rebuild the nix-shell to reflect source code changes, which is seriously hindering workflow.

3. enables python tooling (e.g. the LSP). The python tooling needs to be able to find the imported python packages.

There are some examples, but none are satisfying all 3 requirements:

• the template uses mkPoetryApplication for the developer shell, which violates 2..
• the readme example uses mkPoetryEnv but tooling doesn't find the imported libs, which violates 3..
• the how-to-guide blogpost simply uses poetry itself to manage the imported python packages, and nix just to manage the python interpreter and poetry. This violates 1..

Also they all do things completely different, which is unnecessarily confusing for users.

Also beside the minimalist template example there are no other whole project examples.

My mediocre non-flake solution

I hope this can serve other beginners as a more or less usable template.

Setup

This assumes usage of niv to pin nixpkgs and poetry2nix.

./nix/release.nix

let
  sources = import ./sources.nix;
  pkgs = import sources.nixpkgs { };
  poetry2nix = import sources.poetry2nix { inherit pkgs; };
  my_app = pkgs.callPackage ./build.nix { inherit pkgs poetry2nix; };
in
{
  inherit my_app;
}

./nix/build.nix

{ pkgs
, poetry2nix
}:
let
  python = pkgs.python311;
  projectDir = ./.;

  # just to read name and version from pyproject.toml
  pyProject = (poetry2nix.mkPoetryPackages {
    inherit python projectDir;
  }).pyProject;
  name = pyProject.tool.poetry.name;
  version = pyProject.tool.poetry.version;

  # NOTE: add non python packages, needed at runtime, here
  runtimePackages = with pkgs; [
  ];

  # NOTE: add non python packages, only needed for development, here
  devPackages = with pkgs; [
    poetry
  ];

  # NOTE: add build time dependencies for python packages here, when confronted with
  # ModuleNotFoundError: No module named '...'
  # see https://github.com/nix-community/poetry2nix/blob/master/docs/edgecases.md
  pypkgs-build-requirements = {
    # <package> = [ "<missing build tools>" ];
  };
  p2n-overrides = poetry2nix.defaultPoetryOverrides.extend (final: prev:
    (builtins.mapAttrs
      (package: build-requirements:
        (builtins.getAttr package prev).overridePythonAttrs (old: {
          buildInputs = (old.buildInputs or [ ])
            ++ (builtins.map (pkg: if builtins.isString pkg then builtins.getAttr pkg prev else pkg)
            build-requirements);
        })
      )
      pypkgs-build-requirements)
  );

  build = poetry2nix.mkPoetryApplication {
    inherit python projectDir;
    # NOTE: trade off
    # "rebuild everything from scratch, which can take forever"
    # vs
    # "pull wheels from pypi, when available and accept supply chain attack risks"
    # also necessary, when having errors with `setuptools-rust`
    # preferWheels = true;
    overrides = p2n-overrides;
  };

  dev-env = poetry2nix.mkPoetryEnv {
    inherit python projectDir;
    # NOTE: see above
    # preferWheels = true;
    editablePackageSources = {
      "${name}" = ./src;
    };
    overrides = p2n-overrides;
  };

  shell = pkgs.mkShell {
    # this is the important bit to enable python tooling and thus satisfy all 3 requirements
    # WARNING: the problem is that the python version is hardcoded in this path
    PYTHONPATH = "${dev-env}/lib/python3.11/site-packages";

    packages = [
      dev-env
    ]
    ++ runtimePackages
    ++ devPackages;
  };

  image = pkgs.dockerTools.buildLayeredImage {
    contents = [
      # for debugging the container
      # pkgs.busybox

      build
    ] ++ runtimePackages;
    inherit name;
    tag = version;
    # maxLayers = 100;
    config = {
      Cmd = [
        "/bin/${name}"
      ];
    };
  };
in
{ inherit build shell image; }

./shell.nix

let
  packages = import ./nix/release.nix;
in
packages.my_app.shell

Usage

The first time:

1. nix-shell -p poetry
2. Setup pyproject.toml and poetry.lock: poetry init

Manipulate python dependencies:

1. enter dev shell: nix-shell (or use direnv)
2. poetry <verb> <package>@<version>
3. re enter shell: exit followed by nix-shell

The dev shell has the pinned python version, poetry version and all python packages from the poetry lock. The python packages are also in PYTHONPATH, which should be picked up by your dev tooling.

Build app:

• nix-build -A build

Build image:

• nix-build -A image

Bump version:

• change version pyproject.toml, this will change it in the build and the image.