feat(build): added support for nix flake

[ChetanBhasin]

I don't know if nix installation is desired, but it was very helpful. I would like to leave this here so the maintainers can decide if they would like to have it.

[iamgio]

Hi @ChetanBhasin, thank you for your contribution. I see you build Quarkdown via shadowJar, but that won't be enough for distribution purposes. distZip should be used (see #68).

I'm not familiar with Nix, what's the execution context of flake.nix? What's the difference from distributing via CI?

[ChetanBhasin]

HI @iamgio! Thanks for the quick reply. It's been a few years since I've worked actively with JVM, so I'll have to look into shadowJar vs distZip. I'll look into it, though.

Nix is a few things — a Linux distribution, a hermetic build system (similar to Bazel), and a package manager. In this context, we're using it primarily for package management that could be shipped to either the NixOS (Linux Distro) or any other system that uses Nix package management. Nix allows for separation of dependencies.

Having Nix here gives us the opportunity to distribute it via package manager, and so Quarkdown can be installed using nix profile install ... or via system configuration. This is an example of nix configuration that is portable across machines. Nix also has a very large community, so we really appreciate when we can install and develop something directly. I've written about nix here.

This PR specifically adds a bit more than just the ability to install Quarkdown using nix profile install ... or the configuration based package management though. It also allows someone who is using Nix on their computer to pull Quarkdown and start working on the code without installing other tooling like JVM, Kotlin, etc. because the dev-shell (specified in flake.nix) will provision the environment.

[iamgio]

I'll have to look into shadowJar vs distZip

By default, distZip creates a zip with lib (jars) and bin (scripts). Usually executing a shadow jar or an unzipped dist zip as a user shouldn't have any difference. Quarkdown, however, bundles QMD libraries (such as paper) and HTML documentation (that will be exploited by the wip Language Server) into the dist zip.

Thank you for the explanation!

[alerque]

Thanks for working on a Flake for this @ChetanBhasin. If nothing else this will make it possible for me to add Quarkdown to Polytype.

You might be interested in some discussion in #69 as they apply to the Nix Flake use case as well.

[alerque]

I was able to get the shadowJar build to work just by placing the paper.qmd file at the right place relative to it, but I had to artificially add a segment to the jar path just so that the hard coded ../libs/qmd makes sense. I used:

/usr/share/quarkdown/java/quarkdown.jar
/usr/share/quarkdown/lib/qmd/paper.qmd

Without the extra 'java' segment (which of course I had to account for when making a bin wrapper) the hard coded relative path didn't work. You might try that instead of the distZip dance.

[iamgio]

@alerque ../libs/qmd is the default value, it can be overridden via -l. Though it would still be preferable to use distZip instead of tweaking the shadow jar, which reduces maintainability.

[ChetanBhasin]

@iamgio I've now removed shadowjar in favor of distZip.

[alerque]

@alerque ../libs/qmd is the default value, it can be overridden via -l. Though it would still be preferable to use distZip instead of tweaking the shadow jar, which reduces maintainability.

I'm not "tweaking" the shadow jar, just installing the necessary resources along with it.

The distZip target generates a zip file with a bunch of unwanted pieces as well (e.g. a Windows bat file, but many other bits too). That means to package it, we have to generate the zip, then also have tooling to extract the zip to a temporary location and grab just the bits we want out of it. This makes for extra steps and build time dependencies and just shenanigans compared to just running the build command(s) and copying the right bits to the right places.

A more distro-packaging oriented build target that just generates (only) the right stuff in the first place would probably be the way to go in the long run.

[ChetanBhasin]

@alerque any chance you could test this flake on your end as well if you like it? I've tested on aarch64-darwin and it works just fine. I doubt there should be any issues given that this is a JVM project, but always good to get more people to test things out.

Until it gets merge, one can use nix profile install github:ChetanBhasin/quarkdown#quarkdown.

[iamgio]

@ChetanBhasin I see the version string is hardcoded, I'd rather avoid that. As you can see in build.gradle.kts, the latest version is fetched from the latest v* git tag. This is just helpful for me as I can just run a 2-command script that generates and pushes new tags to create a new release without worrying about misalignment.

You could:

• Fetch the version via gradle (preferred): https://stackoverflow.com/a/48616954
• Fetch it via git as I mentioned. That would bring some unnecessary repetition, so I'd rather go with the first option.

Then it looks good to me, I'd wait for @alerque who definitely knows more about Nix than me. I will just edit the README a bit before merging.

[alerque]

I expected to be able to run the flake straight off the branch, but got an error like so: (Linux x86_64), nix 2.28.3)

$ nix run github:ChetanBhasin/quarkdown -- --help
warning: Cannot parse Nix store ''https://nixpkgs-update.cachix.org''
error: builder for '/nix/store/hfxl7696ir73ivwl6n3w6w5cw6gydicc-quarkdown-1.4.0.drv' failed with exit code 1;
       last 25 log lines:
       > Exception in thread "main" java.net.UnknownHostException: services.gradle.org
       >    at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:572)
       >   at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
       >         at java.base/java.net.Socket.connect(Socket.java:633)
       >   at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304)
       >     at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:178)
       >    at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:533)
       >        at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:638)
       >        at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:266)
       >        at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:380)
       >   at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:193)
       >        at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1257)
       >     at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1143)
       >      at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:179)
       >         at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1702)
       >   at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1626)
       >    at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:224)
       >  at org.gradle.wrapper.Download.downloadInternal(Download.java:100)
       >      at org.gradle.wrapper.Download.download(Download.java:80)
       >       at org.gradle.wrapper.Install$1.call(Install.java:82)
       >   at org.gradle.wrapper.Install$1.call(Install.java:62)
       >   at org.gradle.wrapper.ExclusiveFileAccessManager.access(ExclusiveFileAccessManager.java:69)
       >     at org.gradle.wrapper.Install.createDist(Install.java:62)
       >       at org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:107)
       >         at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:63)
       For full logs, run:
         nix-store -l /nix/store/hfxl7696ir73ivwl6n3w6w5cw6gydicc-quarkdown-1.4.0.drv

[ChetanBhasin]

@alerque it's github:ChetanBhasin/quarkdown#quarkdown. It's not the default package so you have to specify #quarkdown.

[ChetanBhasin]

Actually, nevermind. I did make it the default package. So it's not working then. I'll try to figure out why.

[iamgio]

Also @ChetanBhasin perhaps it's the case to include npm and its puppeteer module as project dependencies, considering what was brought up in #69.

[alerque]

Also @ChetanBhasin perhaps it's the case to include npm and its puppeteer module as project dependencies, considering what was brought up in #69.

Yes, that would be good, it is already usable that way. I've just gotten it working in the AUR recipe, given the system provides puppeteer and a working browser for it, export NODE_PATH="<path to system packages>" in the wrapper before exec java ... quarkdown.jar ... seems to be enough to make it work, I have PDFs to show for it.

I don't know how that will be easy or hard to setup with the way Nix provides puppeteer, but given what you have coming soon anyway in #71 it might as well go in the flake now.

[alerque]

As of 9199fd6 this still does not build for me, it errors trying to download and setup Gradle. ~~I think the approach of trying to let the build fetch it's own Gradle is misguided in the context of Nix Flakes, you should be telling Nix exactly what Gradle to package to supply so that the build is deterministic and reproducible. There is both a gradle and gladle_8 package (currently the same, but it might be smart to use the latter) available in nixpkgs, so this should be pretty easy to overcome. I did notice packaging this for Arch that JDK 24 was a non-starter and used 21 (because it was packaged), but nixpkgs has 23 and 17 on offer.~~

Looking at the Flake you already are trying to supply the Nix packages for Gradle, but somehow the build inputs don't seem to be propagating or something.

[iamgio]

I don't think downloading and setting up Gradle on your own is required. Invoking ./gradlew <subcommand> (not gradle--w stands for wrapper) will already take care of setting up the correct Gradle distribution.

[alerque]

I don't think downloading and setting up Gradle on your own is required. Invoking gradlew (not gradle--w stands for wrapper) will already take care of setting up the correct Gradle distribution.

That's nice for non-developer end users wanting to directly compile from source, but it does not play nice with distro packaging. One is not always able to have network access at all stages of builds, and side-loading things breaks reproducibility tests, etc. It's particularly antithetical to the way Nix works.

[ChetanBhasin]

I agree with @alerque. That's kind of the point of nix is to have hermetic builds. It's only truly reproducible if the hashes actually match.

[iamgio]

What's the state now? @ChetanBhasin @alerque

[alerque]

As of yet this flake has still not worked for me at all. @ChetanBhasin has something going on taken care of locally that isn't coded up in the Flake yet. I'll be sure and mention when it does work for me, but it shouldn't be too hard to reproduce on any system with nix. Once the Flake is right it should be pretty robust for anybody to use, but so far it evidently doesn't wire everything up.

$ nix run github:ChetanBhasin/quarkdown -- --help
warning: Cannot parse Nix store ''https://nixpkgs-update.cachix.org''
error: builder for '/nix/store/35kyb4bnb3z39skqwj3llj7clfq7aw9l-quarkdown-1.4.0.drv' failed with exit code 1;
       last 25 log lines:
       > To honour the JVM settings for this build a single-use Daemon process will be forked. For more on this, please refer to https://docs.gradle.org/8.14.1/userguide/gradle_daemon.html#sec:disabling_the_daemon in the Gradle documentation.
       > Daemon will be stopped at the end of the build
       >
       > FAILURE: Build failed with an exception.
       >
       > * Where:
       > Settings file '/build/source/settings.gradle.kts' line: 1
       >
       > * What went wrong:
       > Plugin [id: 'org.gradle.toolchains.foojay-resolver-convention', version: '0.8.0'] was not found in any of the following sources:
       >
       > - Gradle Core Plugins (plugin is not in 'org.gradle' namespace)
       > - Included Builds (No included builds contain this plugin)
       > - Plugin Repositories (could not resolve plugin artifact 'org.gradle.toolchains.foojay-resolver-convention:org.gradle.toolchains.foojay-resolver-convention.gradle.plugin:0.8.0')
       >   Searched in the following repositories:
       >     Gradle Central Plugin Repository
       >
       > * Try:
       > > Run with --stacktrace option to get the stack trace.
       > > Run with --info or --debug option to get more log output.
       > > Run with --scan to get full insights.
       > > Get more help at https://help.gradle.org.
       >
       > BUILD FAILED in 12s
       >
       For full logs, run:
         nix-store -l /nix/store/35kyb4bnb3z39skqwj3llj7clfq7aw9l-quarkdown-1.4.0.drv
$ nix-store -l /nix/store/35kyb4bnb3z39skqwj3llj7clfq7aw9l-quarkdown-1.4.0.drv
Running phase: unpackPhase
@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking source archive /nix/store/a09lcbr6d9zb7l4gq4vz0m0jgjp9arzr-source
source root is source
Running phase: patchPhase
@nix { "action": "setPhase", "phase": "patchPhase" }
Running phase: updateAutotoolsGnuConfigScriptsPhase
@nix { "action": "setPhase", "phase": "updateAutotoolsGnuConfigScriptsPhase" }
Running phase: configurePhase
@nix { "action": "setPhase", "phase": "configurePhase" }
Running phase: buildPhase
@nix { "action": "setPhase", "phase": "buildPhase" }
Building Quarkdown distribution with Gradle...

Welcome to Gradle 8.14.1!

Here are the highlights of this release:
 - Java 24 support
 - GraalVM Native Image toolchain selection
 - Enhancements to test reporting
 - Build Authoring improvements

For more details see https://docs.gradle.org/8.14.1/release-notes.html

To honour the JVM settings for this build a single-use Daemon process will be forked. For more on this, please refer to https://docs.gradle.org/8.14.1/userguide/gradle_daemon.html#sec:disabling_the_daemon in the Gradle documentation.
Daemon will be stopped at the end of the build

FAILURE: Build failed with an exception.

* Where:
Settings file '/build/source/settings.gradle.kts' line: 1

* What went wrong:
Plugin [id: 'org.gradle.toolchains.foojay-resolver-convention', version: '0.8.0'] was not found in any of the following sources:

- Gradle Core Plugins (plugin is not in 'org.gradle' namespace)
- Included Builds (No included builds contain this plugin)
- Plugin Repositories (could not resolve plugin artifact 'org.gradle.toolchains.foojay-resolver-convention:org.gradle.toolchains.foojay-resolver-convention.gradle.plugin:0.8.0')
  Searched in the following repositories:
    Gradle Central Plugin Repository

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.

BUILD FAILED in 12s

A CI job should probably be setup too so changes to the project build system always get reflected in the Flake any time it requires updates.

[ChetanBhasin]

Hey guys! Sorry I haven't had much time to look into this with my day job taking most of the attention right now, but I'll try to find time for this soon and also setup CI.

[alerque]

I sent a PR to your form Chetan that should at least get you started on a CI test for this.

[iamgio]

I can see a possible problem though: Gradle fetches the latest version from git tags for reasons I explained in a previous comment. It might be problematic to run offline, as the result version would be 0.0.0. I'm not sure how to tackle this -- I'd rather edit as few files as possible whenever publishing a new release.

[alerque]

I can see a possible problem though: Gradle fetches the latest version from git tags for reasons I explained in a previous comment. It might be problematic to run offline, as the result version would be 0.0.0. I'm not sure how to tackle this -- I'd rather edit as few files as possible whenever publishing a new release.

We can have the Flake derive the version from the Git context, whether local or remote. It shouldn't have to be hard coded in the flake at all.

If there is a Gradle command for outputting the version the same way you are deriving it and/or a place we can look in the built sources to extract it we can use that, otherwise some iteration of git describe to get a version string.

[iamgio]

If there is a Gradle command for outputting the version the same way you are deriving it and/or a place we can look in the built sources to extract it we can use that

Two options:

• Adding a new Gradle task to build.gradle.kts, to be invoked via gradle printVersion -q (-q stands for quiet to prevent any additional logging):

  tasks.register("printVersion") {
      doLast {
          println(project.version)
      }
  }
  

• gradle properties -q | awk '/^version:/ {print $2}'

The first option is supposed to be more robust.

[iamgio]

I've just added the printVersion task to the main branch, as my comment above.

[iamgio]

@ChetanBhasin with 1.6.0, Quarkdown becomes package-manager-friendlier as it's not attempting root-only operations such as installing and linking npm packages at runtime.

If you're still interested in the nix flake, you might find a good build example in the Homebrew formula.

[alerque]

I have the hello-world example at least spun up and working on Polytype via local testing, but the PR for it that will update the website via CI is waiting for a working Flake.

[iamgio]

@ChetanBhasin if you're not interested in this PR anymore and nobody else is willing to get in charge, I'm going to close it in 3 days.