Nix devShell and Nix-based GitHub Actions build using `nix develop`

[msgilligan]

This PR adds a flake.nix (and flake.lock) file that can be used to create a Nix development shell for Sparrow. For interactive use (assuming the user has Nix installed and flakes enabled) the shell can be started with:

nix develop

The GitHub Actions package.yaml file has been extended with a job that does a full build of the Sparrow jpackage using the following build command:

nix develop -c gradle jpackage

The flake.lock file makes a snapshot of all the dependencies and makes the build somewhat reproducible (dependencies are still loaded via Maven.)

The flake.nix file could be extended with a full, reproducible Nix build that would run in a sandbox with no network access and no access to undeclared objects in the file system. But this is a good first step towards that.

[msgilligan]

Here is the successful run: https://github.com/msgilligan/sparrow/actions/runs/13511693733

[craigraw]

This is cool, but I am struggling a bit with the "why?" on this, since it appears the major benefit of using Nix is reproducibility, which Sparrow already has. It appears on the surface to be a more complex build setup.

[msgilligan]

This is cool, but I am struggling a bit with the "why?" on this, since it appears the major benefit of using Nix is reproducibility, which Sparrow already has. It appears on the surface to be a more complex build setup.

It's reproducibility "ab ovo" -- all the way back to the beginning. Similar to how Bitcoin Core uses GUIX. It's often called "bootstrappable" to distinguish from the reproducibility that Sparrow (and bitcoinj) currently have.

See:

• https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-building-from-source-all-the-way-down/
• https://luj.fr/blog/is-nixos-truly-reproducible.html
• https://reproducible.nixos.org
• https://reproducible-builds.org
• And, of course, the famous Ken Thompson paper.

It's a goal of mine to be able to verify build from source on secp-jdk and bitcoinj all the way back to the "first" C compiler. And I would like to help move Sparrow in this direction, too. I view this as a long-term project (that will likely take several years.) But each step along the way IMO will increase quality and trustworthiness.

A short term goal for me is to just be able to install Sparrow via Nixpkgs. The current Sparrow in Nixpkgs is x86_64-linux only: https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/applications/blockchains/sparrow/default.nix#L285. A long term goal is to be able to run Sparrow on a Linux Computer running NixOS with every piece of software in the system built with bootstrappable reproducibility.

We've made progress on bitcoinj, but have a way to go. Our base module now only depends on the JDK itself and only two jars (slf4japi and a jar for nullability annotations), the secp-jdk -api module only depends on JSpecify (the new/best solution for nullability annotation.)

I know that Sparrow is a much more complicated beast, but I believe we can work on it one-step-at-a-time without too much impact on the rest of the Sparrow project.

I'm fairly new to Nix, but I am focused on applying it to building the TEE enclave software that I am working on. I've also been applying it to bitcoinj + secp-jdk as a "side-project". It's also easier for me to apply it to Java (than Rust) because I am much more familiar with the build tools.

I find there are many other benefits to using Nix/Nixpkgs/NixOS, though the learning curve is steep. So, if you're at all interested, I'm happy to send PRs that provide and improve a Nix-based build and spend some time helping you get up to speed. This would all be in parallel to the existing build process, although at some point in the future if the Nix-build is looking good and wins your trust you could certainly switch over. This is the same approach I'm using for secp-jdk and bitcoinj. sec-jdk will hopefully be released into production with a Nix-based build, but this is not guaranteed. bitcoinj has yet to merge any Nix code onto its master branch, but I'll likely create a PR very similar to this one that is very limited in scope and I hope Andreas will approve it. (The current WIP PR is to broad in scope and should be reproduced to just the devShell approach for a first step.)

If you're not interested and/or want to postpone until later, I'll understand. I've got plenty of fish to fry. But I do like the idea of a "full stack" build that goes all the way from OS to libraries to desktop app. Let me know what you think!

[msgilligan]

BTW, one of the best introductions to Nix is Section 2 of the original GUIX white paper: https://arxiv.org/abs/1305.4584 -- In about 1 page it explains the big picture very well.

[craigraw]

Thanks for the explanation :) Clearly I have much to learn about Nix.

I know that Sparrow is a much more complicated beast, but I believe we can work on it one-step-at-a-time without too much impact on the rest of the Sparrow project.

In line with this, does it make sense to put the nix_jpackage job into it's own yaml file - say nix-jpackage.yaml? That way it can be triggered independently, and as I don't think there are any artifacts uploaded by the nix_jpackage job, it makes it simpler to manage as an independent unit.

Is it possible to perform a Windows build of Sparrow using this approach?

With respect to merging, I'll need to wait until Sparrow migrates to JDK 23 before merging this to avoid confusion.

[msgilligan]

Thanks for the explanation :) Clearly I have much to learn about Nix.

A good way to get started with Nix, and to learn about it on your own time is to install and use Home Manager: https://github.com/nix-community/home-manager

You can use it on Ubuntu/Debian or on macOS and in its simplest form can be used as a config file that lists packages to install via Nixpkgs. It is especially useful if you want to share configuration between systems. On macOS it co-exists well with HomeBrew and SDKMAN! and on Linux with apt and SDKMAN!.

does it make sense to put the nix_jpackage job into it's own yaml file - say nix-jpackage.yaml?

Yes. I will revise this PR to do that.

Is it possible to perform a Windows build of Sparrow using this approach?

I haven't done it, yet. But I am game for trying.

I'm sure it's possible using cross-platform build tools. I'm not sure what the state of jpackage is these days. Can you run jpackage on Linux to make a Windows exe?

With respect to merging, I'll need to wait until Sparrow migrates to JDK 23 before merging this to avoid confusion.

Of course.

[msgilligan]

@craigraw I made the suggested change and made sure it ran correctly: https://github.com/msgilligan/sparrow/actions/runs/13529101842

(forgive my hack to make it run, but the "Nix Package" workflow wasn't showing up for me in the GUI, so I temporarily added a 'on push' trigger and then removed it)

[craigraw]

On reflection can nix-jpackage.yaml be renamed to package-nix.yaml and the job name renamed to

    name: build-nix (${{ matrix.os }})

[msgilligan]

On reflection can [the file] be renamed ... and the job name renamed...

@craigraw Done. Rebased and force-pushed.