Support for fully custom `:load_from` for escripts

[technusm1]

Hi, I'm trying to use a single rust function inside my elixir app. I followed the setup instructions according to mix rustler.new and successfully got it working under iex. Unfortunately, I was getting an error when doing the same for escript.build, which is when I found out that rustler uses priv directory which is not accessible to escripts. Luckily, I came across the load_from configuration option and tried it in my module like this:

use Rustler, otp_app: :fizzbuzz, crate: "fizzbuzz", load_from: {:fizzbuzz, "/Users/maheep/Downloads/native/libfizzbuzz"}

Unfortunately, all it does is append the path to Application.app_dir output. Here's the problematic line in rustler.ex:

load_path =
          otp_app
          |> Application.app_dir(path)
          |> to_charlist()

which I changed to:

load_path =
          path
          |> to_charlist()

Doing this change allows me to specify fully custom load_from paths and gets things working for me.

MY QUESTION: Is load_from the correct option to use in my case? If so, do I have any other alternative (I don't think its a good idea to mess with a dependency's code).

MY OS: macOS 13.4.1 Ventura

[evnu]

If I understand your question correctly, you are wondering if you should be able to use an absolute path for load_from because currently it only allows a relative path?

[technusm1]

@evnu that is correct. And I also have the question that since I got absolute paths working by changing 2 lines in code, did I just find a bug or is there a reason as to why only relative paths are allowed?

[evnu]

did I just find a bug or is there a reason as to why only relative paths are allowed?

I think this might be a bug, load_from should IMO allow to load from any path. Now, I don't know if we should have a breaking change here, as users might already depend on the handling we have in place right now. So I'd propose that we fallback to Application.app_dir/2 if the path looks like a relative path:

absolute? = Path.absname(path) == path

load_path =
  if absolute? do
    to_charlist(path)
  else
    otp_path |> Application.app_dir(path) |> to_charlist()
  end

We could also add a separate option load_from_abspath which does not do this conversion. That might be a bit nicer with regards to documentation, as we could then document how load_from and load_from_abspath each retrieve the library.

@filmor thoughts?

[technusm1]

I support having a separate option for the sake of backward compatibility and for simplifying things in code. Just check if the compiled artifact is present in absolute path, and proceed accordingly. 😄

[evnu]

Current state here: @filmor and I discussed a possible solution (simply allowing absolute paths) in #558. The solution has pros (simple) and cons (versioning of the library, deployment of the library, etc). @filmor proposed to implement @on_load manually, which is a workaround that works for the use case here. That means that a user cannot use Rustler directly, but replicate that logic and replace the parts that are required for loading the library.

[adamcstephens]

I'm building an application which uses a library built on rustler (ex_git). Because of nix's network sandbox, one cannot rely on the elixir-driven compilation of the rust library.

When I tried using load_from to point at an absolute path of the built library, I was unfortunately met with a similar issue. I've worked around this for now by symlinking the library into place, but I figured I'd share another use case.

ex_git>   'Failed to load NIF library: \'/build/hex-source-ex_git-0.11.0/_build/prod/lib/ex_git/nix/store/0fkzv7r63zi4p08rpsx71wkbnfg0i44l-ex_git_rustler-0.11.0.so: cannot open shared object
 file: No such file or directory\''}}

[filmor]

Nix can compile other Rust projects as well, right? Rustler's mix integration is basically just a wrapper around cargo build in the right directory and then placing the file in priv. If you give cargo all files that it needs up front, it will work without networking, even when run through mix.

[adamcstephens]

The problem is that cargo was trying to download the dependencies for the library. If you have to pre-vendor the dependencies, it's just as easy to use nix to build the rust library first and provide the output to the resulting app.

I'm open to alternatives, but this does work.

In elixir config set

config :ex_git, ExGit, skip_compilation?: true

then for nix

  mixNixDeps = import ./mix.nix {
    inherit lib beamPackages;
    overrides = _: prev: {
      ex_git = let
        name = "ex_git";
        version = "0.11.0";

        src = beamPackages.fetchHex {
          pkg = "${name}";
          version = "${version}";
          sha256 = "1lri3xvslkz8m2f65jfkfmqf9b5jjr5r5r865hwlll5bm316s4ck";
        };

        exgitRustler = rustPlatform.buildRustPackage {
          pname = "ex_git_rustler";
          inherit version;

          nativeBuildInputs = [
            pkg-config
          ];

          buildInputs = [
            openssl
          ];

          src = "${src}/native/exgit";
          cargoHash = "sha256-H2dNNrrz+fc4h7YwLVkyumHTpb5Z3koZ2RwRY2OU3EY=";
        };
      in
        beamPackages.buildMix {
          inherit name version src;

          beamDeps = [prev.rustler];

          appConfigPath = ../config;

          postBuild = ''
            rm priv/native/libexgit.so
            ln -s ${exgitRustler}/lib/libexgit.so priv/native/libexgit.so
          '';
        };
    };
  };