Show how to use `opam-nix` to add dependencies

[brendanzab]

opam-nix looks like a neat tool to read Opam files, and pull down dependencies. Thanks @balsoft for letting me know about it! It would be cool to demo how this could be used in this repo.

It would be good to consider if it's still a good idea to let Dune drive the generation of hello.opam. It's already kind of janky how I check that the generated file is up to date, requiring people to re-run dune build in the dev shell to keep it up-to date. I'd want to ensure that the process of adding a new dependency, then re-building the package is smooth. Perhaps we should just get rid of the generate_opam_files and package specification and let Opam drive pulling down the packages?

[brendanzab]

I guess one of the issues I have is that I don't know how regenerate the opam file without calling dune build. But if I need to add a new dependency, and that dependency will be added by using the Opam file, then I'd assume the initial dune build would fail to run?

[balsoft]

I guess one of the issues I have is that I don't know how regenerate the opam file without calling dune build

I think dune build hello.opam should work and not require any dependencies.

As for the duplication issue, I think it can be approached from the other direction. IIUC we can generate hello.opam using just dune and ocaml without any networking required, so we can do so on the fly using IFD, and then use the existing opam-nix functionality to call the resulting opam package. I'll see if this is as easy as I think it is tomorrow (hopefully)

[balsoft]

Actually, couldn't restrain myself and did it today: https://github.com/tweag/opam-nix/commit/178c26e14388a9af798dd9cae5a88ff461ff9be1

If you remove hello.opam from this repository, then (buildDuneProject "hello" ./. {}).hello builds it successfully.

[brendanzab]

Oh, thanks so much, that was fast! Yeah removing hello.opam might work nicely. I'll probably also want to add some instructions for a raw Opam+Dune workflow, as I want to support that as well - not everyone uses Nix and I want it to be usable with the traditional tooling people know.

So yeah, I've been trying to understand how to use opam-nix but I'm finding the API and examples a bit hard to wrap my head around? I don't remember having the same level of difficulties with naersk (even if some things still confused me at first).

I've jotted down some of my first impressions – trying to capture my clueless, beginner state. They're a bit raw but if you're up for it I'm happy to post them either here, or in private, but I'm aware that I lack lots of context and don't know and your existing constraints/challenges with the project.

[balsoft]

Please do! opam-nix is still being developed, and I'll update documentation as I do so. For now, I think the example for buildDuneProject should be relevant for you.

[brendanzab]

Ah yeah I was more stuggling actually getting to a state where I could call buildDuneProject.

At first I just tried adding opam-nix.url = "github:tweag/opam-nix" and then calling:

{
  checks = {
    hello = (opam-nix.buildDuneProject "hello" ocaml-src {
      doCheck = true;
    }).hello;
  };
}

But yeah of course that didn't work because I'd misread the example 😅. So I tried to look for some full examples.

This is my attempt at trying to get something going based on the README (apologies for the trainwreck):

• Looking at the README I'm intially looking for an example.
• There's a list of them which is nice! I think it might be better if it was in the readme itself, but I can live.
• When I click on the first example, labelled in the readme as "Building a package from opam-repository" it looks very different from what I'm used to from the flake setup, which is initially disorienting.
• I'm not sure what the parameters are for, so I skimmed past those.
• For some reason opam-nix = inputs.self.lib.${pkgs.system}; is being bound - like I'm confused why the system packages are being called opam-nix?
• What is scope, and why is it being returned from the nix file?
• I'm also confused as to what is being shown in the example. Where is the package being built, as was claimed in the name of the example?

Looking through the other examples was similarly confusing, so instead I tried looking at the templates for more help.

• First I looked at the simple template. Simple sounds good!
• The inputs seem pretty standard
• I'm confused why only [ "x86_64-linux" ] is iterated over - wouldn't it be better to use eachDefaultSystem?
• Why are we using legacyPackages in the flake?
• The let binding is rather large and it was hard to understand how it was scoped, especially with the two attribute sets passed to opam-nix.queryToScope (which perhaps I'm not used to reading).
• Again we have opam-nix = inputs.opam-nix.lib.${system}; - for some reason we are not using opam-nix directly, that was bound in the parameter of the outputs, instead acessing it with inputs.opam-nix?
• We have the scope thing again - I'm not sure what that is for. What is a query, and why am I converting it to a scope?
• Assigning the package to null as a way to get the current version seems uhh, ok (if a little ugly), but why do I need to do that when it should be reading the version from my opam file?
• Being able to set the base compiler seems pretty neat!
• setting the default package seems reasonable, still seems kind of weird to be using legacyPackages - perhaps this is something I'm unaware of as a newcomer to nix?

I'm very confused about what this is meant to do:

let
  # ...
  overlay = self: super:
    {
      # Your overrides go here
    };
in scope.overrideScope' overlay;

why the overrideScope'? is there an older deprecated version of overrideScope?

I tried to go on, then reading the concepts in the README, in an attempt to answer some confusions I had, but I was already kind of reaching my limit in terms of information overload.

[brendanzab]

[balsoft]

Examples were originally intended for prototyping the interface and testing the capabilities of the builder, and only then they became actual examples. I have refactored them to be flakes, so it should be a bit easier to understand.

I'm confused why only [ "x86_64-linux" ] is iterated over - wouldn't it be better to use eachDefaultSystem?

For now, only Linux x86_64 is supported, so I only iterate on that. In the future, I may also support macos, but for now there's no point in providing outputs that don't work.

Assigning the package to null as a way to get the current version seems uhh, ok (if a little ugly), but why do I need to do that when it should be reading the version from my opam file?

It was using an outdated API, I have updated the local template. Now the resulting scope will automatically include latest versions of all packages from your project. As for the simple template, it reads the entire opam-repository and so you have to tell it which packages from there you want to resolve for. There's no single "opam file" to read the versions from.

Why are we using legacyPackages in the flake?

legacyPackages is the the only correct output format for nixpkgs scopes. It allows you to nicely build individual dependencies of your package, with the correct switched versions.

[balsoft]

And, as for your example, try

{
  checks = {
    hello = (opam-nix.lib.${system}.buildDuneProject "hello" ocaml-src { }).hello.overrideAttrs;
  };
}

[brendanzab]

For now, only Linux x86_64 is supported, so I only iterate on that. In the future, I may also support macos, but for now there's no point in providing outputs that don't work.

Ahh, I am on macOS - will that mean I'll need to wait for that to be supported?

[brendanzab]

Btw, checked out the examples and yeah, they seem a bit more approachable now which is super nice! :)

[balsoft]

Ahh, I am on macOS - will that mean I'll need to wait for that to be supported?

Yep. I don't have a mac device to test on. I imagine it wouldn't be too difficult to support, basically need to remove some things which were hardcoded to linux and add a depexts map for macos.

[balsoft]

@brendanzab Hey, now it should somewhat work on macOS! External dependency handling is not as good as with Linux yet, but you can help by updating https://github.com/tweag/opam-nix/blob/main/overlays/external/homebrew.nix .

[brendanzab]

Hey, just wondering, do you have any idea how to install packages off Opam that aren't in the nix packages repository? Would I have to invoke opam manually for that?

[balsoft]

The entire idea of opam-nix is to build packages which aren't in nixpkgs :)

[brendanzab]

Ohh - I was still getting about dependencies not being installed when using buildDuneProject so I was a bit confused maybe 🤔

For example (on another project I was experimenting with), I had:

        packages.garden =
          (opam-nix.lib.${system}.buildDuneProject { } "garden" ocaml-src {
            garden = null;
            ocaml-base-compiler = null;
          }).garden;

With this error:

5 |  (libraries yocaml yocaml_yaml yocaml_markdown yocaml_unix))
                                                   ^^^^^^^^^^^
Error: Library "yocaml_unix" not found.
Hint: try:
  dune external-lib-deps --missing --no-config --root . --ignore-promoted-rules --default-target @install --always-show-command-line --promote-install-files false --promote-install-files --release --only-packages garden -p garden --profile release -j 10 @install

[brendanzab]

Another question: when I use buildDuneProject, either with ocaml-base-compiler = null or not, it seems to went to build the OCaml compiler from scratch on different projects, and it doesn't seem like there is even any local caching of compiler builds. Is this normal and expected behavior?

[brendanzab]

One thing that might be interesting to add to the opam-nix repository is something comparing and contrasting it to the existing stuff that nix provides for building OCaml stuff. I.e.

• https://nixos.org/manual/nixpkgs/stable/#sec-language-ocaml
• https://nixos.wiki/wiki/OCaml

[brendanzab]

Another thing I'm wondering about is how I should be setting up my editor. At the moment I'm using vscode-direnv to pick up my nix config in my editor, but I'm not sure how to get merlin to find the build directory of the flake. Should I just accept that I need to run something like dune build --watch --terminal-persistence=clear-on-rebuild separately in a devShell?

[balsoft]

it seems to went to build the OCaml compiler from scratch on different projects, and it doesn't seem like there is even any local caching of compiler builds

This means that the compilers it builds are different for some reason. You can find out the compiler store path by looking at nix show-derivation for both of your projects, and then looking for ocaml-base-compiler in the output. You can then call nix-diff /nix/store/...-ocaml-base-compiler.drv /nix/store/....-ocaml-base-compiler.drv (you may need to nix-shell -p nix-diff first). Perhaps the resolved compiler versions are different, or some of your packages explicitly require some other ocaml compiler.

Another thing I'm wondering about is how I should be setting up my editor

I think adding inputsFrom = [ self.defaultPackage.${system} ] to your mkShell call in the devShell and then having use flake in your .envrc should work.

As for the build error, could you share the generated .opam file for your project?

[brendanzab]

Ahh, I'm thinking it might be related to the yocaml packages not being deployed to opam I added some pin-depends in a .opam.template, but it still seems to fail, so I made an issue: https://github.com/tweag/opam-nix/issues/1

Thanks for the additional pointers, I'll try that stuff a bit later!

[brendanzab]

I think adding inputsFrom = [ self.defaultPackage.${system} ] to your mkShell call in the devShell and then having use flake in your .envrc should work.

Do you have a good idea about how I might get a compatible version of merlin? Currently I get an error like this in VS Code:

/nix/store/hz7rwhn13kk7yxsjrpxw2qg6rqv6s5jj-ocaml-4.13.1/lib/ocaml/stdlib.cmi seems to be compiled with a version of OCaml that is not supported by Merlin.

[balsoft]

One idea would be to add merlin to the packageset (by adding merlin = null to the query) and then using merlin from the resulting scope in your devShell. It sounds a bit hacky, but it is the opam-nix equivalent of installing merlin with opam install merlin, so I think it's fitting.

[brendanzab]

Is there a way of scoping that import to the development shell only?

[balsoft]

Not sure I understand the question.

[brendanzab]

Like, instead of adding merlin as part of the main package, add merlin to the package after the fact?

devShell = legacyPackages.mkShell {
  nativeBuildInputs = [
    legacyPackages.fswatch # for `dune build --watch --terminal-persistence=clear-on-rebuild`
    # legacyPackages.ocamlformat # FIXME: fails to build `uunf` on aarch64-darwin :(
  ];
  inputsFrom = [
    self.defaultPackage.${system} # <--- Add `merlin` and `opam-lsp` here somehow?
  ];
};

[brendanzab]

Or can I set up an OCaml toolchain beforehand, and use that to build my package, and then to setup my devShell? Rather than doing it all in one step?

[brendanzab]

oxalica/rustoverlay separates out the choice of toolchain into an earlier step, which seems like it makes this kind of thing easier? But I dunno.

[balsoft]

Ah, no, I think you misunderstood me?

I'm proposing something like this:

{
  outputs = { self, opam-nix, nixpkgs }: {
    legacyPackages.x86_64-linux =
      opam-nix.lib.x86_64-linux.buildOpamProject { } ./. {
        # This will just ask opam-nix to include merlin in the same scope (package set) as our package, but not necessarily add it as a dependency!
        # This means merlin will be built using the same compiler version etc.
        merlin = null;
      };
    defaultPackage.x86_64-linux = self.legacyPackages.x86_64-linux.my-package;
    devShell.x86_64-linux = with nixpkgs.legacyPackages.x86_64-linux;
      mkShell {
        buildInputs = [
          fswatch
          self.legacyPackages.x86_64-linux.merlin # Note how merlin is separate from my-package. If you remove this line, merlin won't be in the devShell
        ];
        inputsFrom = [
          self.legacyPackages.x86_64-linux.my-package # And my-package doesn't actually depend on merlin at all
        ];
      };
  };
}

[balsoft]

FYI, I have changed the interface a bit: now buildOpamProject accepts a name and only builds one package from the project, and the old buildOpamProject is now buildOpamProject'