std icon indicating copy to clipboard operation
std copied to clipboard

Published 20 hours ago • verified publisher icon divnix

Std `ops.mkStandardOCI` compatibility problem with `ops.mkOperable` to wrap

Open Pegasust opened this issue 1 year ago • 3 comments

Fixed in https://github.com/divnix/std/pull/331

I have a pretty standard poetry2nix web application that I wrap on its environment derivation using hypercorn. The built OCI fails to detect the correct binary location because it relies on lib.getExe. mkOperable works as expected, though.

Symptoms

I don't have the exact error that throws from Docker engine. It should be something along the line of Docker container not being to start up because /bin/entrypoint not found.

If you have access to ls (I use Python's os.listdir because I'm too lazy to build a dev layer), You would see that /bin/entrypoint exists, but it is a broken symlink.

$ std //ops/oci/p2n-env-oci:load
docker-daemon:<image>:<hash>
$ docker run -it --rm --entrypoint /bin/sh <image>:<hash>
$ python3

import os

def follow_symlinks(path):
    if os.path.islink(path):
        link_path = os.readlink(path)
        print(f'{path} -> {link_path}')
        return follow_symlinks(link_path)
    elif os.path.exists(path):
        print(f'Path is a real file or directory: {path}')
        return path
    else:
        print(f'Path not found: {path}')
        return None

follow_symlinks('/bin/entrypoint')

One example output is

/bin/entrypoint -> /nix/store/<hash>-oci-setup-links/bin/entrypoint
/nix/store/<hash>-oci-setup-links/bin/entrypoint -> /nix/store/<hash>-operable-python3-3.10.11-env/bin/operable-python3
Path not found: /nix/store/<hash>-operable-python3-3.10.11-env/bin/operable-python3
>>> os.listdir('/nix/store/<hash>-operable-python3-3.10.11-env/bin/')
["operable-python3-3.10.11-env"]

Workaround

  p2n-env-fix = workspace_app.dependencyEnv.overrideAttrs (prev: {
    # HACK: This allows mkOperable and mkStandardOCI to resolve symlink correctly
    name = "p2n-env-fix";
    meta.mainProgram = "python3";
  });

Nix flakes and its workaround variant

# nix/dev/packages/tools.nix
{
  inputs,
  cell,
}: let
  system = inputs.nixpkgs.system;
  poetry2nix = inputs.nix-boost.pkgs.${system}.mypkgs.poetry2nix;
  workspace_root = "${inputs.self}";
  workspace_app = poetry2nix.mkPoetryApplication {
    projectDir = workspace_root;
  };
in {
  p2n-env = workspace_app.dependencyEnv;
  p2n-env-fix = workspace_app.dependencyEnv.overrideAttrs (prev: {
    # HACK: This allows mkOperable and mkStandardOCI to resolve symlink correctly
    name = "p2n-env-fix";
    meta.mainProgram = "python3";
  });
  frontend = {}; # omitted
}

# nix/ops/exe.nix
{
  inputs,
  cell,
}: let
  inherit (inputs.std.lib) ops;
  runtimeScript = ''
      hypercorn my.backend:app --bind '0.0.0.0:10140' --worker-class uvloop
  '';
  runtimeEnv = {
    DIST_LOC = "${inputs.cells.dev.packages.frontend}/dist";
  };
in {
  p2n-env-ops = ops.mkOperable {
    package = inputs.cells.dev.packages.p2n-env;
    runtimeInputs = [
      inputs.cells.dev.packages.p2n-env
      inputs.cells.dev.packages.frontend
    ];
    inherit runtimeScript runtimeEnv;
  };

  p2n-env-fix-ops = ops.mkOperable {
    package = inputs.cells.dev.packages.p2n-env-fix;
    runtimeInputs = [
      inputs.cells.dev.packages.p2n-env-fix
      inputs.cells.dev.packages.frontend
    ];
    inherit runtimeScript runtimeEnv;
  };
}

# nix/ops/oci.nix
{
  inputs,
  cell,
}: let
  inherit (inputs.std.lib) ops;
  pyproject = builtins.fromTOML (builtins.readFile "${inputs.self}/pyproject.toml");
  ws-authors = pyproject.tool.poetry.authors;
in {
  p2n-env-oci = ops.mkStandardOCI {
    name = "p2n-env-oci";
    # NOTE: swap to p2n-env-fix-ops fixes it
    operable = inputs.cells.ops.exe.p2n-env-ops;
    meta = {
      # NOTE: unfair, std doesn't allow me to give more tags :(
      # including this and `nix` and `std` tells me to remove either `meta.tags` or `tag`
      # tags = ["ops-latest" "ops-${ws-version}" "${ws-name}-${ws-version}"];
    };
    config = {
      ExposedPorts."10140/tcp" = {};
      Volumes = {
        # For secrets and env
        "/var/run" = {};
      };
      Env = [
        "ENV_PATH=/var/run/.env"
        "GNMI_CONFIG_LOC=/var/run/.config.yml"
      ];
      Labels."org.opencontainers.image.title" = "p2n-env";
      Labels."org.opencontainers.image.authors" = builtins.concatStringsSep ", " ws-authors;
    };
  };

  # Currently, there is no good documented way to publish to arbitrary repositories at once
  # NOTE: untested
  p2n-env-gl = cell.oci.p2n-env-oci.overrideAttrs {
    meta.repo = "gitlab.<domain>.com/<group>/<repo>/<image>";
  };
}

Some more insights from build artifacts

nix-repl> packages.aarch64-darwin.p2n-env.name
"python3-3.10.11-env"

nix-repl> packages.aarch64-darwin.p2n-env.meta.name
"python3-3.10.11"

nix-repl> lib.getExe packages.aarch64-darwin.p2n-env-ops
"/nix/store/<hash>-operable-python3-3.10.11-env/bin/operable-python3"

Thoughts

Requests

  • Resolve this issue along with https://github.com/divnix/std/issues/299 by adding minimal build recipes along with troubleshoots
  • There is a bit of rough edges mentioned in the provided workspace for https://github.com/divnix/std/issues/251 in the case of tags
  • Add check phase for mkStandardOCI that checks for symlink integrity. Might be doable at the phase of writing /bin/entrypoint and before using nix2container to write .json manifest

Pegasust avatar Jun 26 '23 17:06 Pegasust

Ran into this same issue today as well, but not with Python.

It looks like if the package uses pname/version (vs. just name) and thus has a name of ${pname}-${version} the entrypoint generated script can't find it as it drops the version.

I think it's related to the fact that the operable wraps by using ${package.name} at one point, but I got a little lost in the weeds trying to trace it thru the various blockTypes and functions.

jboyens avatar Jul 12 '23 18:07 jboyens

Sorry for the delay in looking into this and thanks for the report!

The issue, so to speak lies in this line and derives indeed from the implementation choices of getExe:

https://github.com/divnix/std/blob/301a649b81276a0a11832b81a09956d0838da7ed/src/lib/ops/mkOCI.nix#L48

# including this and nixandstdtells me to remove eithermeta.tagsortag``

This has fortunately just been fixed.

For my own better understanding:

  runtimeScript = ''
      hypercorn my.backend:app --bind '0.0.0.0:10140' --worker-class uvloop
  '';

Which of the code in the flake brings my.backend:app into the python path? (I'm not 100% familiar, atm).

Add check phase for mkStandardOCI that checks for symlink integrity. Might be doable at the phase of writing /bin/entrypoint and before using nix2container to write .json manifest

That is a good idea, however it also seems to me that this python case could have maybe a better designed interface?

Just ideas, but why not e.g. mkStandardOCIWithHypercorn? For my taste # nix/ops/oci.nix could be an even shorter file. Would creating a library of tool-specific mkStandardOCI-interfaces with all the hard learned expriences already codified be a good idea?

I'm musing about this because currently, the operable's package is not even put into the PATH environmet, but I can see how different runtime enviroenments would like to see that package made available in their relevant environment, whether that be PATH or PYTHONPATH or anything else. It would be somewhat scoped to scripted languages.

blaggacao avatar Jul 13 '23 22:07 blaggacao

Context on hypercorn + poetry2nix

The part that brings in (I guess $PYTHONPATH, or allow for python -m my.backend) is workspace_app.dependencyEnv, which is

  workspace_app = poetry2nix.mkPoetryApplication {
    projectDir = workspace_root;
  }.dependencyEnv;

and pyproject.toml that uses poetry. Poetry2nix takes care of taking these dependencies and either build them, or acts as Nix .whl broker via poetry.lock for preferWheels = true builds.

# minimal pyproject.toml
[tool.poetry]
name = "acme-monolith"

[tool.poetry.dependencies]
python = "^3.10"
databases = "^0.7"
hypercorn = "^0.14.3"
uvloop = "^0.17.0"

# This requires the path for `./my/backend.py:app` or `./my/backend/__init__.py:app`
# Though, out-of-the-box top-level package detection from `poetry` should cover most 
# cases, I'm putting this here for explicitness
[[tool.poetry.packages]]
include = "my"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

For context/archival purposes, .dependencyEnv sort of just calls inputs.nixpkgs.python.mkEnv, which calls inputs.nixpkgs.buildEnv or something similar then calls it a day, so the original package name is not passed through to .dependencyEnv.

I'm unsure which specific part injects onto PYTHONPATH, by inspecting the built artifacts, it looks like the python executable is wrapped

Framework-specific recipes

As @jboyens pointed out, this issue is unfortunately not specific to ${poetryApplication}.dependencyEnv, but is arose from some common patterns when it comes to interpreted languages, so this issue could probably be addressed via common pattern like mkStandardOCIFromInterpretedEnvironment, though feel free to suggest shorter names.

If there are some framework-specific nitpicks, I do think there's novelty in keeping mkStandardOCI having a low-level interface with some basic default configuration that covers 80%+ cases, with integrations having their own modules or transformed like (lib.dev.mkNixago lib.cfg.lefthook) into something like (mkStandardOCIWith oci.interpretedEnv), and add some recipe docs, linking to this issue. My only concern is at some point, our API change, and the recipe docs effectively points to some out-dated way to solve this

Other

# including this and nix and std tells me to remove either meta.tags or tag

This has fortunately just been fixed.

Thanks for this!

Also taking a look on https://github.com/divnix/std/pull/331

Pegasust avatar Jul 14 '23 05:07 Pegasust