how to mimic _tags file within myocamlbuild.ml

[agarwal]

Within myocamlbuild.ml, how can I apply some tags to files matching a filter.

I know the recommended way of doing this is to define an _tags file, but I don't want to because:

• It's just one more file cluttering my source directory.
• I want to use the information that would go into it in other places also. For instance, I'd like to define a rule to generate a .merlin file. Thus, I need the list of packages my project uses within myocamlbuild.ml.
• I'd rather just use OCaml as my configuration language.

I tried mutating Options.tags, but I'm not sure that's the right variable to change and I don't know how to do this selectively on files matching a pattern.

[gasche]

The internal function to do this is Configuration.parse_{string,file}, but it is not exposed from signatures.mli so that would be Ocamlbuild_pack material. You could also mutate Options.tag_lines which can start with a prefix: you would need to check but I think that -tag-line "<foo>: bar,baz" works.

[agarwal]

Here's a test case:

$ cat myocamlbuild.ml 
open Ocamlbuild_plugin

let () = dispatch (function
  | Before_options -> (
    Options.use_ocamlfind := true
    ;Ocamlbuild_pack.Configuration.parse_string "<*>: foo"
  )
  | _ -> ()
)

$ touch a.ml

$ ocamlbuild a.cmo

Now looking at _build/_log, I can see that tag foo did get applied to each target. However, various changes to this file make this stop working. Where's the right place to call Configuration.parse_string. I tried lots of places, and in a real myocamlbuild.ml, I end up not seeing the tag applied anywhere.

(The extensive use of mutable variables in ocamlbuild isn't warming me to it.)

[agarwal]

Ignore last question. I was doing something else wrong. Here's a solution that is working so far:

• Copy contents of your _tags file into an OCaml variable in myocamlbuild, and delete _tags.
• Call Configuration.parse_file in your Before_options hook.

Leaving open for now, until I test a bit more.

[gasche]

Either Before_options or After_options should be fine. (Other places should work or not depending on the kind of tags you add this way; if you add a traverse tag after the build directory has been slurped, it's too late.)

(The extensive use of mutable variables in ocamlbuild isn't warming me to it.)

Pull requests to improve the current status would be warmly welcome.

I think your use-case is interesting and would warrant improvements to the current OCamlbuild API. Please keep providing feedback on what you would need.

One related feature that was proposed (I can't find the mantis ticket back) is to provide a filesystem overlay: some kind of master _files file that would contain a hierarchy (for example as a YAML file) mimicking a filesystem hierarchy, where you would put the content of what currently is put in _tags or foo.mllib or bar.itarget files. At the time of reading/slurping the project, OCamlbuild would walk both the filesystem and this file -- we could also add programmatic way to add things to the virtual project filesystem for your use-case. This would allow people to have ocamlbuild-specific files cluttering their repository, in a rather flexible way.

It looks like such a feature could be useful for your purposes. Would you be interested in helping implement it?

[agarwal]

Thanks for the additional input.

some kind of master _files

My goal is to eliminate _tags, mllib, odocl, etc. I know OCaml, so I want to express my build in OCaml. Adding a YAML file seems to me the wrong direction.

Would you be interested in helping implement it?

I'm not yet convinced that ocamlbuild has a good model. The general problem is rather complex and interesting, and has use cases beyond compiling code. I'm still exploring, but will contribute what I can to ocamlbuild along the way (if not code, at least feedback and examples of anything that works for me).

[gasche]

My proposal is to explicit the fact that OCamlbuild works on a virtual abstraction of the "project tree", not only the filesystem sub-hierarchy at hand. Currently this virtual project tree is a mirror of the fragment of the filesystem delimited by traverse flags (and partially transformed by include directives), but we could and should ways to union this with other representations. For most users (think of those that are complaining about having to touch myocamlbuild.ml and want shoehorn every feature in _tags), a YAML file (or whatever file format that conveniently expresses filesystem overlays) would be a more convenient way to describe an additional project overlay than a direct API, and you want a direct API, but both interfaces are complementary and would be served by the same implementation work.

[agarwal]

a virtual abstraction of the "project tree"

What can you specify with such a tree? Does it add any expressivity beyond what you can currently do, which is dependent on builtin conventions for all these specific files: _tags, .mllib, etc? IIUC, this would only let you specify the content of these files in a different place, which perhaps you find more convenient.

I'm kind of considering the same thing. I'm now expressing my _tags file as a string within myocamlbuild.ml. However, now I can construct the string programmatically, so it's really more powerful.

think of those that are complaining about having to touch myocamlbuild.ml and want shoehorn every feature in _tags

Is it clear why they prefer _tags? I wonder if that's because writing a myocamlbuild.ml file has been so scary. A nicer user API might remove that issue.

both interfaces are complementary

Would users of the OCaml API benefit at all from the overlay proposal? If not, then they aren't complementary, but rather the OCaml API is the more fundamental one and the overlay proposal is an easier interface for simpler cases (but as above I wonder if the OCaml API couldn't be made just as easy).

Is this the usual debate between making a new configuration language or providing an embedded DSL. The latter is always better if your users don't have an aversion to the host language, which in this case they shouldn't; ocamlbuild users are OCaml programmers.

[gasche]

IIUC, this would only let you specify the content of these files in a different place, which perhaps you find more convenient.

Yes (but if you have suggestion for "more expressive" things, why not). Basically, instead of creating foo/bar.mllib with content Bla\nBli, you could add

foo:
    bar.mllib:
        Bla
        Bli

to _ocamlbuild_files, or call

Overlay.add_file ["foo"; "bar.mllib"] {|
  Bla
  Bli
|}

from myocamlbuild.ml.

The interest in having this avaibility is that some users dislike having to create many files in their project to satisfy the build system. For example, people ask for the ability to use _tags to specify global options that affect ocamlbuild's semantics (such as -use-ocamlfind), while that does not fit the _tags model (which is justd supposed to add tags to targets). If I proposed to add a _conf file specifically for this purpose, many user would find this inconvenient. If I propose a single file in which to put the ocamlbuild-specific information, they may be happier.

Is it clear why they prefer _tags? I wonder if that's because writing a myocamlbuild.ml file has been so scary. A nicer user API might remove that issue.

I tend to agree with you -- the very first step is to have a documented API and this what this manual is about. But I suspect that some users would still be ready to jump through hoops to not write OCaml code -- it would be nice to find a way to please both crowds. (I resist adding to much to _tags if it means diverging from its current simple declarative semantics, which is also why other files could be considered.)

Would users of the OCaml API benefit at all from the overlay proposal?

Well the API would let you add and change things in the project tree overlay. But it may also give you more fine-grained, better typed, better ways to do things. This is similar to what you are doing right now: feeding strings to a parsing function is not really ideal but it is expressive, I'm sure you would appreciate a typeful API to build tag descriptions. (It would be rather easy to add one with one pull request exposing the type lexer.mli:conf to end-users.)

[agarwal]

Your examples dissuade me further from the overlay idea. It seems to be adding cruft on top of cruft. You'll have yet another way to specify something, making it harder to understand a project's build rules by human readers.

The _tags file at least has some reasonable amount of information, but files like mllib and odocl are so basic that I feel having a file for them is too much. Why not just give a string list to some function in myocamlbuild.ml and be done with it. And now I can, if I want to, generate that list by reading my directory contents. I really can't imagine not wanting that.

On a separate note, there are modules Tags, Configuration, and Lexers, all of which are fundamentally about tags. The names are a bit frustrating. (I'll post such issues to the new ocamlbuild repo when it is created.)

[agarwal]

If I propose a single file in which to put the ocamlbuild-specific information, they may be happier.

Make that file be myocamlbuild.ml.

I'm sure you would appreciate a typeful API to build tag descriptions

Yes and surprised to not find it.