More configurable remote mercurial server

[wraithm]

Right now, it seems that the remote mercurial repositories are hardcoded to have "hg" as the host with "hg" as a path. It would be nice if this host and path combination were configurable.

In the module Remote_repo_path,

let family t =
  match t with
  | Ssh { host = "hg"; path } ->
    (match List.map (Abspath.to_list path) ~f:File_name.to_string with
     | "hg" :: family :: _ -> Some family
     | _ -> None)
  | File _ | Ssh _ -> None
;;```

[dwang20151005]

It would be easy to accept any host, and harder to accept any layout of repositories on the disk. Would it work for your use case if we allowed any host but the path part still had to look like /hg/FAMILY/... ? E.g., could you use symlinks to recreate the layout we expect?

[wraithm]

Yeah. I could probably make that work.

Another question, I'm running into issues with not having a .hg/bookmarks.current. My guess is that this is related to your active_bookmarks extension. I see that you have a reference to some internal extension in the hgrc in this repo. Googling didn't turn up much. If these extensions are public, could you point me to them? If not, is there any way that you could put the mercurial extensions that iron needs somewhere, perhaps in this repository?

[dwang20151005]

@mkoconnor will know more about this than I do.

Are you setting up an external fe instance? That's exciting!

[wraithm]

Nevermind. I might be off about my hunch. Mercurial eventually made a bookmarks.current when I hg upped around. Really, the problems I'm having are that the iron server is constantly stuck in "waiting for hydra" mode for any feature. The fe tools wait-for-hydra command says the reason for "waiting for hydra" is Expecting_bookmark_update. Maybe I just have a bad mental model of what "waiting for hydra" means. I've tried pushing to the remote repo, and that doesn't seem to trigger anything. Basically, I don't know what "waiting for hydra" means or how to properly get a feature ready for review.

I would like to set up an external fe instance if this all works out :) fe looks like a great tool! Right now, I'm doing this all on my laptop.

[dwang20151005]

The fe server expects to work with a build daemon. Ours is called hydra. But basically you need something that will watch for your commits, update a working directory to that commit, and call env BOOKMARK=... fe internal hydra-worker.

You also need to set up obligations in your repo, to define who needs to review what. Take a look at the setup in one of our automated tests: tests/sample-repos/two-users. Note that the files committed as dot-fe.sexp need to be renamed to .fe.sexp before you run the hydra worker. (We store the test .fe.sexp files under a different name so that these test files don't get picked up by our production fe instance.)

It's worth pointing out that fe is probably overkill for personal use.

[wraithm]

I see. Right now, I have a mercurial pretxnchangegroup hook to test/build stuff when I push to the remote repo. Is there a way to have iron work with that? Or must I have a daemon watching? The hydra-worker code is a little difficult for me to understand what's going on. Would it make sense to make some sort of abstraction around hydra?

I did set up the .fe.sexp file similarly to the two-users example. I made it one user, and I also was able to successfully add myself as a user. There seemed to be errors on the iron server when I ran BOOKMARK=testfeature fe internal hydra-worker.

Thanks for pointing that out :) I intend to use iron for business.

[dwang20151005]

I've never done it that way, but an hg hook would probably work, at least for small repositories. At our scale, the hydra-worker will peg a core for ~2m, but I imagine your users will inform you when the latency becomes inconvenient. :-)

Indeed, the hydra worker subcommand is designed for the calling conventions of our build daemon. I've noted in our issue tracker that we should make it more conventional.

Can you show me these errors, and the output of fe obligations check in your test repo, and the output of the hydra worker?

[wraithm]

Output of fe obligations check:

$ fe obligations check
("Error loading .fe/obligations-repo.sexp file at repo root"
 /path/to/irontests/testrepo
 (Unix.Unix_error "No such file or directory" open
  "((filename /path/to/irontests/testrepo/.fe/obligations-repo.sexp) (mode (O_RDONLY O_CLOEXEC)) (perm 0o0))"))

A .fe folder was never made. I only have the .fe.sexp.

If I run, BOOKMARK=testrepo fe internal hydra-worker, I get this output on the server:

2017-02-14 14:04:20.075894-06:00 Info (hydra-worker((uuid a9d02867-e412-311d-c268-8e0bc026fa0c)(by mwraith)(at((human_readable(2017-02-14 14:04:20.073603-06:00))(int_ns_since_epoch 1_487_102_660_073_603_000)))(hostname machinename)(machine_zone America/Chicago)(executable fe)(executable_version NO_VERSION_UTIL)(action((feature_path testrepo)(rev_zero((human_readable(0))(node_hash aa459acee01edde234626511e1b927751c94575a)))(tip(((human_readable())(node_hash bb57c0ee744293ef540aa3da265f99e8f3f7a5b3)))))))((query_uuid a9d02867-e412-311d-c268-8e0bc026fa0c)(server_received_query_at((human_readable(2017-02-14 14:04:20.075826-06:00))(int_ns_since_epoch 1_487_102_660_075_826_000)))(server_computed_reaction_at((human_readable(2017-02-14 14:04:20.075855-06:00))(int_ns_since_epoch 1_487_102_660_075_855_000)))(reaction(Ok((base"0 [aa459acee01e]")(feature_id 9ddab27e-2558-3bff-dcf8-1fffad25628d)(need_diff4s_starting_from())(aliases())(lines_required_to_separate_ddiff_hunks 0)(worker_cache((worker_obligations())(worker_rev_facts())(properties((max_size 1)(status Disabled)(max_items_per_rpc 1)))))))))) -- [what: query]
2017-02-14 14:04:20.710641-06:00 Info (update-bookmark((uuid 84a69df1-2f12-3d8d-43d8-9e8dca8d74c3)(by mwraith)(at((human_readable(2017-02-14 14:04:20.710367-06:00))(int_ns_since_epoch 1_487_102_660_710_367_000)))(hostname machinename)(machine_zone America/Chicago)(executable fe)(executable_version NO_VERSION_UTIL)(action((feature_path testrepo)(feature_id 9ddab27e-2558-3bff-dcf8-1fffad25628d)(info(Error(monitor.ml.Error("Process.run failed"((prog xargs)(args(-d"\n"-r grep --binary-files=without-match -s ^<<<<<< -l))(working_dir /path/to/irontests/testrepo)(exit_status(Exit_non_zero 1))(stdout"")(stderr("xargs: illegal option -- d""usage: xargs [-0opt] [-E eofstr] [-I replstr [-R replacements]] [-J replstr]""             [-L number] [-n number [-x]] [-P maxprocs] [-s size]""             [utility [argument ...]]"""))))("Called from file \"src/deferred1.ml\", line 20, characters 40-45""Called from file \"src/job_queue.ml\", line 159, characters 6-47"))))(augment_worker_cache((worker_obligations())(worker_rev_facts()))))))((query_uuid 84a69df1-2f12-3d8d-43d8-9e8dca8d74c3)(server_received_query_at((human_readable(2017-02-14 14:04:20.710591-06:00))(int_ns_since_epoch 1_487_102_660_710_591_000)))(server_computed_reaction_at((human_readable(2017-02-14 14:04:20.710616-06:00))(int_ns_since_epoch 1_487_102_660_710_616_000)))(reaction(Ok())))) -- [what: query]

It's complaining about invalid options to xargs.

After running fe internal hydra-worker, fe list says report Iron bug instead of wait for hydra,

$ fe list testrepo -display-ascii

feature	lines	next step
testrepo	pending	report Iron bug

[dwang20151005]

The obligations error is easy enough: you need to create the .fe directory. There is one in the sample repo to mimic.

As for the hydra-worker, what version of xargs do you use? It looks like we shell out assuming a different set of options than is supported by the xargs available on your machine.

FWIW, my xargs --version gives

xargs (GNU findutils) 4.4.2 Copyright (C) 2007 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

Written by Eric B. Decker, James Youngman, and Kevin Dalley. Built using GNU gnulib version e5573b1bad88bfabcda181b9e0125fb0c52b7d3b

How hard would it be for you install that version on your system? I don't know if we would be willing to support multiple versions of xargs for external users. At the very least, I'd have to ask someone about that.

[wraithm]

I'm running this on macOS, so I'm using BSD xargs. I can just set up a Linux virtual machine for GNU findutils. The build server that I would have iron run on is Linux anyways, so might as well do that.