Cache dependencies for clojure projects using Tools.deps.alpha in addition to Leiningen and boot

[armincerf]

Currently Netlify will only cache dependencies in Clojure projects if there is a valid project.clj or build.boot file, which are used by the Leiningen and Boot build tools.

However Clojure now has an official API for dependency management and uses a deps.edn file rather than the project.clj or build.boot files used in older build tools.

It makes sense to add support for deps.edn files as this is the official way to manage dependencies in Clojure now.

Relevant PR https://github.com/netlify/build-image/pull/117/files

[kennyjwilli]

Are there any known workarounds for the time being?

[armincerf]

The only one I can think off is maintaining a project.clj as well, but that's a bit of a pain

[armincerf]

Actually just looked at the code and seems like it doesn't actually matter what's in the project.clj file, just that it exists. I'll test with an empty project.clj to make sure tomorrow but if that works then it will be easy to check for project.clj or deps.edn as they will both be cached in the same way.

I will make a PR when I get around to it if that's the case.

[kennyjwilli]

I think that will work well for most cases. Our build currently eats up ~2.5 mins of downloading deps on every build haha.

The one exception is that deps.edn allows for Git deps which are stored in ~/.gitlibs (by default. See docs).

[minikomi]

A completely empty (touch project.clj) project.clj causes lein to try and run it, which results in an error.

[mainej]

I believe this would be fixed by #553.

[kennyjwilli]

Hi @mainej. Thanks for working on that PR. The Clojure CLI supports Git Deps. I'd suggest adding the ~/.gitlibs directory to the cached set of dirs (see docs).

[mainej]

@kennyjwilli Good suggestion. 👍 I've added caching of ~/.gitlibs to the PR and verified that it works by using the instructions for testing builds with the cache.

[barrelltech]

~What is the status of this? I would like to use netlify but am building a project with the clojure cli~

Nevermind, it's been merged, there was just an issue in my build script 😂 I reckon this issue can be closed! :)

[mainej]

@bbarrell this issue shouldn't be closed. It isn't fixed.

As I believe you discovered, Netlify does install the clojure cli for building deps.edn based projects (though see caveats below).

However, 485 was opened to address a different problem—build times of Clojure projects are unnecessarily slow, because ~/.m2 and other directories aren't cached at the end of a build.

(Note... what follows is addressed to Netlify, not you.)

Because of this and other bugs, it can be hard to build Clojure projects on Netlify. Netlify isn't paying attention to these issues, nor have they offered guidance or workarounds. It's a shame because I would really like Netlify to be an easy place to deploy Clojure projects. As you can see below, I and several other people have tried reporting issues, creating PRs, or looking for other solutions, but we've gotten very little response.

By my accounting there are two important outstanding bugs related to Clojure:

1. PR 484: the default clojure cli is very outdated (Netlify's default version of the CLI was built Nov 25, 2019). Other tooling, including shadow-cljs expects newer versions of the CLI.
2. Issue 485, this issue: builds do not cache dependencies. Would be fixed by PR 553.

I tried to write a "build plugin"—Netlify's mechanism for extending the build process—to work around the caching issue, but was never able to get ~/.m2 cached. I filed an issue about that, but never got any response from Netlify. Discouraged by that experiment, I've never tried writing a build plugin that would install an updated version of the Clojure CLI. I doubt it would work, because I think that install needs sudo, which you don't have in the plugins, AFAIK. The build plugin infrastructure is cool, but it doesn't address some of the fundamental needs that language communities need to build their projects.

Anyway, if you've gotten your project to build @bbarrell, nice! It is possible. Cross your fingers that it keeps working!

[mainej]

The caching fix in #553 was closed because it wasn't for focal, the current version of Ubuntu targeted by Netlify. I've re-opened a simplified version of #553 as #831, based off the focal branch.

#484 has also been closed, for the same reason. It could technically be re-opened against focal, but the workaround is another option for getting an updated Clojure.