Add a failing test for sites plugins with two plugins

[shym]

This PR is a kind of bug report: it adds a test that is a small variation on the example in the manual with two plugins. One of those plugins comes with a test that really depends on this plugin (so it is expected to succeed whether the other plugin is available or not). It shows that a test will fail if it launches the application before all the plugins have been fully built, even if they are not required.

I suppose the root cause is the fact that the META files for the plugins are installed before the .cmxs are built.

We came across this problem with ocaml-ci failing from time to time in a project containing various plugins but with tests that always require only one of them.

[Alizter]

We usually put tests about sites in the otherlibs/dune_site/test.

[shym]

I moved it accordingly (and rebased it). Is that a proper way to report the corresponding bug, by the way?

[Alizter]

Writing a PR demonstrating the bug is more than enough. You might need to sanitize the test for it to work on macos. Best bet is greping for a common string and maybe you can include the full exception as a comment if needed.

[Alizter]

cc @bobot

[shym]

I added a grep that will hopefully work across platforms :crossed_fingers: But CI now fails on macos on a test this PR does not modify ?!?

[Alizter]

@shym Thanks. The failing test occasionally happens out of the blue, nothing to do with you. It might be a little while before we get round to merging this, probably when we have a fix. You are more than welcome to try to fix this.

[shym]

Thank you @Alizter for your advice!

I gave it some thoughts but I’m not sure what would be the best way to tackle this. The two possible solutions I’ve contemplated (but not tried yet, I didn’t figure out how to add the dependencies) are:

• the rule to generate a META file such as plugin1/META.plugin1 could depend on the various .cma, etc. files it refers to (is a META file of any use before those files are built? :thinking:)
• those dependencies could be added rather on the rule to install the META file, if building the META earlier is interesting.

I think these two options would ensure a build from a clean state works but I’m not convinced about incremental builds though. In a situation where the not-needed-by-the-test plugin must be updated, its META file would already be present (I think it would not be removed early just thanks to the added dependencies, would it?) and the test executable would try to load the plugin while it’s being updated...

[bobot]

The problem is indeed a problem of dependencies but more at the level of packages. The fact that dune-sites look into the locally installed files without taking into account the dependencies of the rule is the problem. Sandboxing all the installed files was(/is?) seen as too costly. Dune could tell to the executable which plugins are really available using environment variable.

Perhaps the sandboxing information can contain which locally installed files are really available (because part of dependencies), and the information provided to the executable.

As a workaround, in project that uses plugins (e.g. Frama-C) during tests the plugins are manually activated and the dependency to the corresponding package specified. frama-c -no-autoload-plugins -load-plugin FOO