ocamlbuild
ocamlbuild copied to clipboard
ocaml
ocamlbuild deletes student's source files (misplaced in _build)
Hello -- I have a recurrent problem with ocamlbuild / OcaIDE. We use this combination in teaching. Several students keep mistakenly placing their source files inside the _build directory. When they attempt to compile, ocamlbuild wipes out their files, and there is no way of recovering. I suppose we could tell the students to use "ocamlbuild -no-hygiene", but this assumes that they listen to us, which they often don't. Is there a foolproof way of ensuring that ocamlbuild does not delete these source files? Thanks, Francois.
Is there a foolproof way of ensuring that ocamlbuild does not delete these source files?
That seems a little bit against the philosophy of the whole thing which is that whatever is under _build can be safely wiped without consequence.
It seems to me we should rather try to understand why they end up writing their files under the _build directory. Is this maybe a problem with ocalIDE (don't know about this). It seems I can remember that it did happen to me under some circonstances using emacs. So is there maybe a particular setup that makes editors jump to errors into _build rather than in the source tree ? IIRC the trick used by ocamlbuild to report errors in the source tree is to copy it in the _build directory and then cwd there to run the build instructions so that errors are still reported as if they were in the source tree.
Another solution to the problem could be that now that 4.03's Unix has support for symlink on windows is to try to symlink the sources rather than copy them there. So that if they start editing the files in the _build tree they would end up in the source tree anyways.
I am not sure if they create their files in _build in the first place, or if they get confused after ocamlbuild copies their files into _build. I think at least in a few cases we saw the former -- nothing was left after _build was cleaned up by ocamlbuild.
Maybe OcaIDE should hide the _build directory; that would be another way of avoiding the problem.
An amusing problem!
Daniel's idea to use symlinks seems reasonable, but we would need to double-check that ocamlbuild is fine with these files being modified behind its back. (For example if somewhere it assumes that it can recompute the file hash and compare it to what's in the source project to know which targets should be rebuild, then we have a problem.)
Another possible idea would be to check the modification time of the source files in _build and fail loudly when we are about to remove one that is more recent than the original source file. I have mixed feeling about using modification times, though; OCamlbuild avoided them so far, and they can be unreliable on some systems. I'm worried it could introduce more problems than it solves.
(Also I find the idea of using -no-hygiene rather dubious, and I'm not even sure it would solve the problem. Would it prevent ocamlbuild from erasing _build/foo.ml when foo.ml has changed and is copied back, for example?)
It seems I can remember that it did happen to me under some circonstances using emacs.
Incidentally this just happened to me. Unfortunately I don't have any kind of diagnosis data to provide (unless there is some kind of hidden emacs log file).
Note that I don't rule out the possibility that I actually opened this file by mistake myself.
I'm sure that's not @fpottier's student case but I have now catched an instance of ocamlbuild directing me to source in the _build directory: ocamldoc error messages.
Steps to reproduce:
cd /tmp
echo "(** hey {ol {1 bla}} *)" > a.mli
echo "A" > doc.odocl
ocamlbuild doc.docdir/index.html
+ /Users/dbuenzli/.opam/4.03.0/bin/ocamldoc.opt -dump a.odoc a.mli
File "/private/tmp/_build/a.mli", line 0, character 11:
hey {ol {1 bla}}
^
File "/private/tmp/_build/a.mli", line 0, character 11:
hey {ol {1 bla}}
^
2 error(s) encountered
Command exited with code 1.
Compilation unsuccessful after building 2 targets (0 cached) in 00:00:00.
I'm not sure what would be a good way to avoid that. The .mli files that ocamldoc depends on are "built" by ocamlbuild, that is found and copied to the _build directory; but in the general case they may of course not exist in the source, be the result of a preprocessing from .mli4 file, or generated by Menhir, etc.
Currently what happens is that .mli are found and copied to the build directory unchanged. One option would be to add a notion of "move hook" to ocamlbuild, or just a separate kind of copy rule that would override the default copy-to-build behavior; and those hooks for OCaml source files would insert lexer directives with the original source file (just like preprocessors would typically do that). But that seems kind of ugly to me. What do you think?
and those hooks for OCaml source files would insert lexer directives with the original source file (just like preprocessors would typically do that). But that seems kind of ugly to me. What do you think?
That's actually something I always considered in the space of solutions along with symlinks. But it also puts me off quite a bit.
In fact I always thought that the clean solution was that the compilers should allow us to define on the cli the file path used for error messages via a -error-file-path option. I never made an issue since I never took the time to consider all the tenants et aboutissants (e.g. are syntax errors spit out by the compiler always about a single source path in a given run).
At a certain point we should have a real look at the compiler options and make sure they allow us to be driven by build system that allow to cleanly compile objects in a directory separate from the source (I didn't follow the recent -o discussions but I know it was also an issue one at some point when multiple files are output).
Yeah, I agree your idea sounds potentially better, and it could be useful in other situations. (For example if the source is taken from the web for some reason, you might want to put an URL there?) I think it shouldn't be terribly hard to implement - we already use a different logic to set the source name when we get a serialized AST from Camlp4.
At a certain point we should have a real look at the compiler options and make sure they allow us to be driven by build system that allow to cleanly compile objects in a directory separate from the source (I didn't follow the recent -o discussions but I know it was also an issue one at some point when multiple files are output).
That is something I discussed with @bobot, having a clean language to specify inputs and outputs, but we never had time to make progress on that. (Having memorable names for warnings instead of just numbers is higher on my TODO list, but also no progress.)
Note that this ocamldoc thing is quite nasty in practice, I just lost a few hours of work because of this.
The thing is that once you start editing .mli in _build if you don't pay attention you may find yourself using the switch-view command of caml-mode which brings you to the implementation corresponding to the mli and voilà you are editing the implementation in _build.
(If you wonder how I can program for a few hours without trying to compile my project, the answer is merlin).
So it seems that the first priority would be to get ocamldoc to display relative instead of absolute file paths, to be consistent with the OCaml compiler (this is why we never noticed the issue with ocaml{c,opt}). I can work on that.
Then there is the idea of inserting lexer directives when copying OCaml source files. Conceptually this is not really a nice change (the idea that target, if found in the source project, are copied unchanged in neat) so I'm a bit hesitant.
Finally, we should wonder if there is a general way to solve the problem of "wiping out user-edited files from the repository". Given that ocamlbuild is supposed to keep track of the files produced in _build (in fact the design allow for under-specification of rules output), and that there is already machinery in place to compute the digest of all build results, it should be possible to implement a prudent mode that would check that the files in built are unchanged from what ocamlbuild remembers. This would slow down the build, though (we would get twice as many digest calculation for each intermediary target, one before and one after production its), and I am not sure that it is a good deal for most of the ocamlbuild users, so deciding whether to enable the option by default or not will be a delicate conversation.
Then there is the idea of inserting lexer directives when copying OCaml source files. Conceptually this is not really a nice change (the idea that target, if found in the source project, are copied unchanged in neat) so I'm a bit hesitant.
I don't think we should go the lexer directive way. What about symlinks ? It seems Windows is becoming friendlier to them (see here).
Why not, indeed. That still does not prevent the problem of editing generated file (only copied source files), but maybe that is good enough?
I have a first patch up against ocamldoc. The change is rather blind-sighted, though; I cannot be fully sure it is correct, and this may delay integration.
@dbuenzli I have pushed a commit to Tuareg that offers to switch to the corresponding .ml/.mli file outside _build if it exists or make the buffer read-only if no corresponding file is found. Let me know what you think.
@fpottier That does not solve your problem as your students do not use Tuareg but something similar could be developed for OcaIDE.
We have done a first round of fixing, but I think that there is a larger usability issue at play here that should be taken seriously. ocamlbuild keeps a cache/log of the build actions it did in the past, so it should in theory be able to detect the presence of exogenous objects in _build (or of an object that has been changed by the user?). What I don't completely understand right now is when it would be appropriate to look for those and warn the user about it (I would be wary of doing this "before each build", because this could have a performance cost, unless we accept to use modification-time instead of hashing; but maybe doing this and making it fast is the only solution).