use :include in install stanza

[timbertson]

I have a package which installs a server program, but also a bunch of resource files (fonts, images, CSS styles etc). I can happily use jbuilder to install the server binary, but I can't figure out how to install the resources without listing them explicitly (not much fun since this includes a large number of files that I didn't write, copied from twitter-bootstrap).

My first thought was that maybe I could write a rule to generate resource-files.sexp and then (:include resource-files.sexp) that in my (files ...) stanza, but that isn't allowed. Is it possible to add this?

[ghost]

I was thinking of allowing globs in (install ...) stanzas. Would that be enough for your case?

[timbertson]

I don't think so - in this case I'd like to:

• include everything under res/ (recursively)
• follow symlinks in this traversal
• drop the res from the destination path (e.g. res/images/foo becomes images/foo)

I could work around any of those being missing (particularly the third, it's just a cosmetic thing), but it doesn't seem like a great fit given that none of them are currently supported by globs.

Aside: it would be pretty neat from an extensibility point of view if pretty much everything supported (:include) - giving an escape hatch which is less severe than generating the whole file. Are you trying to keep the interface simple by not allowing that, or is it difficult / costly to add support each place it's allowed?

[ghost]

(:include) introduces dynamism in the jbuilder build graph, so it can only be used where jbuilder supports it. At the moment, jbuilder doesn't support dynamic targets. In particular, this mean that after reading all the jbuild files, jbuilder is able to answer the following question without running any command: is X a target?.

jbuilder uses install stanzas for two things: generating .install files and maintaining a tree of symlinks under _build/install that is an image of what we'll be installed. The later is used to make it easier to have dependencies that are part of the same workspace. For instance, in jbuild files you can write ${lib:foo:bar} which means file bar of library foo. If library foo is not available in the current workspace then ${lib:foo:bar} will be resolved to a path where foo is installed on the system; for instance /usr/local/lib/foo/bar. Otherwise it will be resolved to something like _build/install/default/lib/foo/bar, which would be a symlink to _build/default/foo/src/bar for instance.

If we allow arbitrary (:include ...) in install stanzas, then we can no longer answer the quesion is _build/install/X a target without running commands.

There is a prototype to extend jbuilder to support dynamic targets, but it's not ready yet.

To come back to your use case, are files under res generated? Otherwise, I think something equivalent to (files_recursively_in X) would work well.

[timbertson]

Ahh, I see. That does make it tricky.

Couldn't you assume that if someone references ${lib:foo:bar} then foo needs to be located, but not go any further than that? In particular, I guess it would mean that any requested file for an external library is assumed to be buildable (and you know its path because you know where foo is to be installed), and it's a runtime error if it fails to exist after installation (since I assume installation is atomic; you wouldn't just install specific files which were needed from external libraries).

To rephrase, I guess what I'm suggesting is to terminate the "is X buildable" at the library layer, not the file layer, since jbuilder can't look inside non-jbuilder libraries anyway. But I have no idea if that would work, so it's up to you whether that's a useful suggestion :)

To come back to your use case, are files under res generated?

Not individually, and not by jbuilder - they're externally-built (npm packages, multiple size variants of input images)

Otherwise, I think something equivalent to (files_recursively_in X) would work well.

Yeah, that would work well for my use case, as long as it followed symlinks (or could be made to). Would it have src and dest prefixes (i.e. "install files from res/ into share/www")? That would be useful, to prevent the source code layout restricting the installed layout.

[ghost]

Sorry, I was away for a while and I forgot to reply.

Couldn't you assume that if someone references ${lib:foo:bar} then foo needs to be located, but not go any further than that?

That's what happens already. However, when you do want to build the file in question, you need to find out what produces it. This can be worked out and I think it eventually this will be possible, but this requires some work on the internals of jbuilder and it's not a quick change.

BTW, until we have a better solution for this issue, with the tip of jbuilder you can do the following:

(include jbuild.inc)

(rule (with-stdout-to jbuild.inc.gen (run ./gen-list-of-files-to-install.exe)))

(alias
 ((name update-jbuild)
  (action (diff jbuild.inc jbuild.inc.gen))))

The jbuild.inc file will be in the source tree (rather than in _build). You'll have to create an empty one initially, then whenever you want to update it, you can do:

$ jbuilder build @update-jbuild --auto-promote

If you only want to check that it is up-to-date, you can do:

$ jbuilder build @update-jbuild

[timbertson]

Thanks for the idea and the diff action! I've (finally ;)) tried this out now, and hit a couple of snags:

Firstly, when I pass --auto-promote the command still exits with 1 when the generated file differs. So I end up having to run:

jbuilder build @src/www/jbuild-res.inc --auto-promote || jbuilder build @src/www/jbuild-res.inc --auto-promote

Which is awkward, but I'm assuming this should be an easy fix.

(I also chose to name the alias the same as the file in question, not sure if that's more or less confusing ;))

But secondly, I can't get my .gen file to be regenerated. If I change some file which affects the list of generated resources (but which isn't ./gen-list-of-files-to-install.exe itself), I can't get jbuilder to ever regenerate the file, even if I ask for it explicitly (with jbuilder build @src/www/jbuild-res.inc.gen). It just doesn't bother running, because none of its known dependencies are out of date. I think this is essentially another case of https://github.com/ocaml/dune/issues/255 (but maybe slightly easier, since I'm explicitly requesting it be built).

Since the whole point of this is to include files which were built outside jbuilder, having jbuilder refuse to regenerate the file makes it difficult. I can at least use an external build tool to just build jbuild-res.inc.gen myself explicitly, but it's quite a dance to get right and make sure the file actually gets regenerated.

[ghost]

For the exit code, the idea is that because the file has changed, you might need to build again to get a fix point.

Running a command every time is a bit complicated as it's not clear what to do once you support polling builds. Maybe the generator could generate the dependencies? For instance the rule that generate the file depends on an alias that you extend in the generated file?

[timbertson]

For the exit code, the idea is that because the file has changed, you might need to build again to get a fix point.

Hmm, fair enough. I still think it should be 0 on --auto-promote though, since if you need to distinguish that case couldn't you just not use --auto-promote? i.e.:

if ! jbuilder build @diff-target; then
  jbuilder build @diff-target --auto-promote
  jbuilder build all-the-other-things
fi

The current workaround is fine though :man_shrugging:

Maybe the generator could generate the dependencies?

Maybe. This particular target would be affected by:

• updating my bootstrap dependency version (this is effectively depending on package.json)
• me manually adding another image into the source location where res/images is generated from (i.e. depend on all files in images/)
• me modifying any of the build scripts which control what paths are created in res/, which currently is:

gup/_build/bootstrap.gup
gup/src/www/res/all.gup
gup/src/www/res/fonts.gup
gup/src/www/res/images.gup

...but if I add to that set or any of these scripts start sourcing other scripts, I'd need to manually add those to the list of dependencies which should cause a rebuild (I have no automated way to get this list).

So it could be done, but it's quite a lot of effort to maintain an accurate dependency list. And if I miss something, the result is an incorrect build which I can't force jbuilder to rebuild. For me, the time wasted by running the command on every build would be well worth the reduced effort in accurately tracking dependencies, since it takes ~0.03s.

[ghost]

BTW, currently you can always use the OCaml syntax as a workaround. The code will be executed on each invocation of jbuilder.

[mseri]

I was discussing with @rgrinberg the possibility to use :include or a glob in the install stanza. For me glob would be enough, I'd be happy to give a shot to the feature if you point me to the right direction

[rgrinberg]

Actually, I think that we should just finish off https://github.com/ocaml/dune/pull/962 first. Let's support static variables first before considering (:include ..). @mseri would you like to finish off that PR, I believe the original author likely abandoned it anyway. Note that (:include ..) is incompatible with the current model of the install dir so it wouldn't be so easy to support