How to use plugin system, trying to use keyrings-google-artifactregistry-auth

[DylanRJohnston-FZ]

Describe the issue

I'm trying to make use of poetry2nix where we have a private artifact repository on google cloud. Normally the keyring plugin keyrings.google-artifactregistry-auth allows Poetry to authenticate against the private repository. However it's unclear how to achieve this with poetry2nix.

Additional context

I initially tried using an overlay to put keyrings-google-artifactregistry-auth in the propogatedBuildInputs but quickly realised this doesn't work because Poetry doesn't use the typical nixpkgs callPackage convention and instead uses poetry itself to manage its own dependencies.

I could potentially achieve this by overriding the pyproject and poetrylock files to include the google keyring plugin.

, pyproject ? projectDir + "/pyproject.toml"
, poetrylock ? projectDir + "/poetry.lock"

However when searching the repository, I found an undocumented withPlugins method that I was hoping you'd help me figure out how to use.

[DylanRJohnston-FZ]

poetry.toml

[repositories]
[repositories.redacted]
url = "https://redacted.pkg.dev/redacted/python"

pyproject.toml

[[tool.poetry.source]]
name = "redacted"
secondary = true
url = "https://redacted.pkg.dev/redacted/python/simple/"

[rszamszur]

I guess withPlugins is experimental/not yet implemented.

The only idea I have at the moment is try to build poetry2nix with custom poetry: https://github.com/nix-community/poetry2nix/blob/3cc82ca6ccca2a06dbb05cb0b74efd5d50e60d48/default.nix#L3 such that would already contain the required plugin. However, I do not know if it would work. Might give a try in spare time.

[MatrixManAtYrService]

The word plugin is often overloaded... Are we talking about poetry plugins like you'd put here?

poetry self add myplugin

If so, maybe a good step would be documenting what is and isn't supported re: plugins in the readme? For instance, my team calls this plugin in a pre-commit hook and I'm not sure if adding it the plugin to the poetry in my devshell is unsupported, or if I'm just overlooking the way to do it.

[DylanRJohnston-FZ]

@rszamszur the option you linked to has been removed from the code. What is the proper way to override the inbuilt poetry derivation now?

[DylanRJohnston-FZ]

The ability to override the poetry package was removed without an appropriate replacement in #1106

[rszamszur]

@DylanRJohnston-FZ It should still be overridable via nixpkgs overlays.

Currently poetry2nix overrides nixpkgs.poetry with its own derivation: https://github.com/nix-community/poetry2nix/blob/02e4a29cb4ec64f2f5e8989084b80951df2bbb64/overlay.nix#L5

I'll do some experiments over the weekend. I'll let you know if I manage to make this work.

[rupurt]

Would love to know how to install plugins. I'm trying to install these 2 plugins.

poetry self add poetry-multiproject-plugin
poetry self add poetry-polylith-plugin

[opeik]

@rszamszur @DylanRJohnston-FZ This worked for my use case:

poetry = pkgs.poetry.overridePythonAttrs (old: {
  propagatedBuildInputs = old.propagatedBuildInputs ++ [pkgs.python3Packages.keyrings-google-artifactregistry-auth];
  catchConflicts = false;
  doCheck = false;
});

Hopefully this helps with the integration 🤞

[DylanRJohnston-FZ]

The magic of propogatedBuildInputs.

[eightysteele]

@DylanRJohnston-FZ i have a potentially silly question...

how do i use this workaround from @opeik to get poetry2nix working with a private artifact on google cloud:

poetry = pkgs.poetry.overridePythonAttrs (old: {
  propagatedBuildInputs = old.propagatedBuildInputs ++ [pkgs.python3Packages.keyrings-google-artifactregistry-auth];
  catchConflicts = false;
  doCheck = false;
});

here's my default.nix so far:

{ pkgs ? import <nixpkgs> { } }:
let
  poetry2nix = pkgs.callPackage (builtins.fetchTarball
    "https://github.com/nix-community/poetry2nix/archive/master.tar.gz") { };
  python312 = pkgs.python312;
  keyrings = pkgs.python312Packages.keyrings-google-artifactregistry-auth;
  gexEnv = poetry2nix.mkPoetryEnv {
    projectDir = ./.;
    editablePackageSources = { gex = ./gex; };
    python = python312;
    pyproject = ./pyproject.toml;
    poetrylock = ./poetry.lock;
    preferWheels = true;
  };
in pkgs.mkShell {
  buildInputs = [ gexEnv pkgs.google-cloud-sdk keyrings ];
  shellHook = ''
    # keyrings are available here and I'm authenticated in google cloud
  '';
}

when i when i add a private artifact in tool.poetry.source and try nix-shell, i get these errors:

building '/nix/store/had00ag5f29zdjwlv5hmmc596cl60sjv-foo-0.4.0-py3-none-any.whl.drv'...
Reading index https://us-west1-python.pkg.dev/proban/prob/simple/foo/
Traceback (most recent call last):
  File "/nix/store/0rzh74dywg10ggl3ds6khsdpc84nbp96-fetch-from-legacy.py", line 82, in <module>
    response = urllib.request.urlopen(req, context=context)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/lib/python3.11/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/lib/python3.11/urllib/request.py", line 525, in open
    response = meth(req, response)
               ^^^^^^^^^^^^^^^^^^^
  File "/nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/lib/python3.11/urllib/request.py", line 634, in http_response
    response = self.parent.error(
               ^^^^^^^^^^^^^^^^^^
  File "/nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/lib/python3.11/urllib/request.py", line 563, in error
    return self._call_chain(*args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/lib/python3.11/urllib/request.py", line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/nix/store/7hnr99nxrd2aw6lghybqdmkckq60j6l9-python3-3.11.9/lib/python3.11/urllib/request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 401: Unauthorized

thanks for any help!