`ImportError` when running a Python package that includes share libraries

[Atry]

Describe the bug Python packages installed from pip cannot find libstdc++.so.6

To reproduce Given the flake.nix

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-22.11";
    systems.url = "github:nix-systems/default";
    devenv.url = "github:cachix/devenv";
  };

  nixConfig = {
    extra-trusted-public-keys = "devenv.cachix.org-1:w1cLUi8dv3hnoSPGAuibQv+f9TZLr6cv/Hm9XgU50cw=";
    extra-substituters = "https://devenv.cachix.org";
  };

  outputs = { self, nixpkgs, devenv, systems, ... } @ inputs:
    let
      forEachSystem = nixpkgs.lib.genAttrs (import systems);
    in
    {
      devShells = forEachSystem
        (system:
          let
            pkgs = nixpkgs.legacyPackages.${system};
          in
          {
            default = devenv.lib.mkShell {
              inherit inputs pkgs;
              modules = [
                {
                  # https://devenv.sh/reference/options/
                  packages = [ pkgs.hello ];

                  enterShell = ''
                    python -m grpc_tools.protoc --help
                  '';

                  languages.python = {
                    enable = true;
                    venv = {
                      enable = true;
                      requirements = ''
                        grpcio-tools
                      '';
                    };
                  };
                }
              ];
            };
          });
    };
}

Run nix develop --impure, then I got the error

$ nix develop --impure
warning: Git tree '/home/atry/devenv-poetry' is dirty
Requirement already satisfied: grpcio-tools in ./.devenv/state/venv/lib/python3.10/site-packages (from -r /nix/store/zl8ljp68qhqs4gd836isanbq6z393arr-requirements.txt (line 1)) (1.56.2)
Requirement already satisfied: grpcio>=1.56.2 in ./.devenv/state/venv/lib/python3.10/site-packages (from grpcio-tools->-r /nix/store/zl8ljp68qhqs4gd836isanbq6z393arr-requirements.txt (line 1)) (1.56.2)
Requirement already satisfied: setuptools in ./.devenv/state/venv/lib/python3.10/site-packages (from grpcio-tools->-r /nix/store/zl8ljp68qhqs4gd836isanbq6z393arr-requirements.txt (line 1)) (65.5.0)
Requirement already satisfied: protobuf<5.0dev,>=4.21.6 in ./.devenv/state/venv/lib/python3.10/site-packages (from grpcio-tools->-r /nix/store/zl8ljp68qhqs4gd836isanbq6z393arr-requirements.txt (line 1)) (4.23.4)

[notice] A new release of pip is available: 23.0.1 -> 23.2.1
[notice] To update, run: pip install --upgrade pip
Traceback (most recent call last):
  File "/nix/store/zn2g96d0hhk5h8x7982m2gbbawgwsrvz-python3-3.10.11/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/nix/store/zn2g96d0hhk5h8x7982m2gbbawgwsrvz-python3-3.10.11/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/atry/devenv-poetry/.devenv/state/venv/lib/python3.10/site-packages/grpc_tools/protoc.py", line 20, in <module>
    from grpc_tools import _protoc_compiler
ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory

Version

e91205acb792f6a49ea53ab04c04fbc2a85039cd

This error can be reproduced on both Ubuntu and NixOS

[Atry]

The following workaround can suppress the error.

packages = [ pkgs.stdenv.cc.cc.lib ]

However, Ubuntu native executables, e.g. /usr/bin/git, do not work with this workaround. It also breaks NixOS executables if they require a version of glibc or libstdc++ different from the project's.

[Atry]

~~I think the best fix for this issue is to create a wrapper of python to set LD_LIBRARY_PATH for Python only and support manylinux ABI out-of-box, instead of setting a project-wise LD_LIBRARY_PATH.~~

LD_LIBRARY_PATH wrapper would break subprocesses created by python.

[Atry]

I created a pull request helping resolving this issue.

• https://github.com/NixOS/nixpkgs/pull/248777
• https://github.com/Atry/nix-ld-so-cache-test

[domenkozar]

I have created a test case here: https://github.com/cachix/devenv/blob/python-rewrite/tests/python-native-libs/devenv.nix

However for some reason it doesn't work still.

❯ python -c "import os;print(os.environ['LD_LIBRARY_PATH'])"
/home/domen/dev/cachix/devenv/.devenv/profile/lib

❯ ls .devenv/profile/lib|grep stdc++
libstdc++.so.6

❯ python -c "from ctypes.util import find_library;print(find_library('stdc++'))"
None

❯ python -c "from ctypes.util import find_library;print(find_library('GL'))"
libGL.so.1

[Atry]

I think LD_LIBRARY_PATH is fragile. Maybe we should not add devenv.packages to LD_LIBRARY_PATH by default? To me, devenv.packages stands for cli tools.

[domenkozar]

How about languages.python.nativeDependencies?

[domenkozar]

I'm thinking about languages.python.libraries = config.packages, so that it matches how other languages work, by picking it up from packages.

That allows anyone with an issue to go and explicitly set it if needed.

If it turns out to be a bad idea, we can remove the default and issue a warning.

[Atry]

Do you mean you would not put config.packages to the shell's LD_LIBRARY_PATH?

[domenkozar]

Exactly, only pass it to python by default, but can be changed.

[Atry]

Sounds good to me

[domenkozar]

https://github.com/NixOS/nixpkgs/issues/7307#issuecomment-1689673234

[alecandido]

Sorry to bump an old thread: this problem persists (also because none of the issue or PR related has been closed or merged).

I made a few trials, and it seems like the workaround above is incompatible with an arbitrary version of Python from cachix/nixpkgs-python. The error is:

ImportError: /nix/store/flf14c3ibr83jsa070j25hg5gjapydhl-glibc-2.37-8/lib/libc.so.6: version `GLIBC_2.38' not found (required by /nix/store/6zl9iqg8hacf5y0h7b4agc5ym0yxzwa8-devenv-profile/lib/libstdc++.so.6)

I guess that version of Python to be using a different glibc version from the one in the stdenv (since it's clearly fishy that glibc-2.37-8 goes looking for glibc 2.38).

Any advice about how to solve this situation?

[Atry]

@AleCandido Have you tried inputs.nixpkgs-python.inputs.nixpkgs.follows = "nixpkgs";

[alecandido]

Thanks @Atry, I obviously didn't.

It recompiled Python and stopped looking for the wrong version of glibc.

[Atry]

I just realized there is already a way to specify fallback library path in glibc's dynamic linker: https://github.com/NixOS/nixpkgs/pull/248547#issuecomment-1995469926

I wonder if we could use LD_AUDIT instead of LD_LIBRARY_PATH to support manylinux ABI?

[Atry]

Here is a demo to solve this issue using LD_AUDIT: https://github.com/Atry/devenv-ld-floxlib

You can compare the solution with https://github.com/Atry/nix-ld-so-cache-test

[domenkozar]

Oh, that's really cool! Would that solve all glibc version incompatibility vues?

[Atry]

Yes. glibc should be solved by python’s RUN_PATH, not ld-floxlib

[Atry]

libstdc++ would be solved by ld-floxlib’s stdenv. Other shared libraries would be solved by FLOX_ENV

[SomeoneSerge]

Here is a demo to solve this issue using LD_AUDIT: https://github.com/Atry/devenv-ld-floxlib

You're fast! It would also be nice to provide ld-floxlib in Nixpkgs. Also we could patch it to reproduce the LD_FALLBACK_PATH interface too

[domenkozar]

~I wish someone wrote this in Rust, to be sure it's safe.~ Not possible due to dynamic linker restrictions

[domenkozar]

@Atry can you make a PR with a proper implementation using floxlib?

[Atry]

@domenkozar Would you mind reviewing other PRs first? https://github.com/cachix/devenv/pull/1018

[jeff-hykin]

can you make a PR with a proper implementation using floxlib?

Just to help with this. My understanding is a general fix is basically changing the python derivation to be like @ Atry's existing flake, plus 0. Adding arguments for floxlib values (like zlib in Arty's code)

1. Using something like https://github.com/FRidh/make-binary-wrapper to always set those ENV vars right before calling python
2. Including floxlib as a runtime dependency
3. Maybe (maybe conditionally) modifying the RUN_PATH of the python executable to include the correct glibc (I'm a bit unclear on this one)

If that doesnt sound far off from reality, I might try it a month from now

[SemMulder]

~I wish someone wrote this in Rust, to be sure it's safe.~ Not possible due to dynamic linker restrictions

@domenkozar can you elaborate on why this isn't possible in Rust? I was planning on giving it a try sometime soon ;).

[Atry]

I don't know why but I tried to create a LD_AUDIT hook using statically linked musl instead of glibc and it always crashes.

On Tue, Apr 30, 2024 at 2:03 AM Sem Mulder @.***> wrote:

I wish someone wrote this in Rust, to be sure it's safe. Not possible due to dynamic linker restrictions

@domenkozar https://github.com/domenkozar can you elaborate on why this isn't possible in Rust? I was planning on giving it a try sometime soon ;).

— Reply to this email directly, view it on GitHub https://github.com/cachix/devenv/issues/773#issuecomment-2084771646, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAES3ORWHIUFT54LIWQXKUDY75M6DAVCNFSM6AAAAAA3E7ZW36VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBUG43TCNRUGY . You are receiving this because you were mentioned.Message ID: @.***>

[SemMulder]

I don't know why but I tried to create a LD_AUDIT hook using statically linked musl instead of glibc and it always crashes.

Might be because musl and glibc are not binary compatible? Per https://www.musl-libc.org/faq.html:

Binary compatibility is much more limited, but it will steadily increase with new versions of musl. At present, some glibc-linked shared libraries can be loaded with musl, but all but the simplest glibc-linked applications will fail if musl is dropped-in in place of /lib/ld-linux.so.2.

[Atry]

I used to think that a statically linked musl can coexist with a dynamically linked glibc.

On Tue, Apr 30, 2024 at 2:56 AM Sem Mulder @.***> wrote:

I don't know why but I tried to create a LD_AUDIT hook using statically linked musl instead of glibc and it always crashes.

Might be because musl and glibc are not binary compatible? Per https://www.musl-libc.org/faq.html:

Binary compatibility is much more limited, but it will steadily increase with new versions of musl. At present, some glibc-linked shared libraries can be loaded with musl, but all but the simplest glibc-linked applications will fail if musl is dropped-in in place of /lib/ld-linux.so.2.

— Reply to this email directly, view it on GitHub https://github.com/cachix/devenv/issues/773#issuecomment-2084871280, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAES3OQBORWJ2UYSGZUEGG3Y75TFFAVCNFSM6AAAAAA3E7ZW36VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBUHA3TCMRYGA . You are receiving this because you were mentioned.Message ID: @.***>

[jeff-hykin]

I can't say if the ABI would cause problems, but I can say a statically linked musl with a dynamically linked glibc sounds like a cursed runtime

[Atry]

@Atry can you make a PR with a proper implementation using floxlib?

@domenkozar Do you think https://github.com/DDoSolitary/ld-audit-prefer-runpath would be a better solution, given that a lot of existing softwares are hard-coded to set of get LD_LIBRARY_PATH, thus it would not be feasible to replace all LD_LIBRARY_PATH usage with FLOX_ENV?