Enable git bisect with multi-repos with gitdist and TriBITS

[bartlettroscoe]

Blocked By: ???

Description:

Since TriBITS archives <Project>RepoVersion.txt files with every CI and Nighty on CDash it was easy enough to manually go back in history and reproduce any build to allow a type of manual bisect. (This has allowed CASL VERA to reproduce any build shown on CDash with history going back months). But I appreciate the need/desire for using the automated git bisect process with git bisect run <script>. Therefore, I have been thinking about how to address the git bisect problem for compatible sets of repos with gitdist for a few months now.

The basic idea is that you would commit the <Project>SubRepoVersion.txt file (as automatically produced by the TriBITS configure process or generated manually using gitdist) to the base <Project> git repo whenever you have a known working compatible set of repos (more on how and when to commit that file below). That is a simple as:

$ cd <projectDir>/
$ cp <build-dir>/<Project>SubRepoVersion.txt .
$ git add <Project>SubRepoVersion.txt
$ git commit

You could see the commits in the base repo where this file got updated as simple as:

$ git log -- <Project>SubRepoVersion.txt

That shows you what commits that git bisect can consider.

Then, you would git bisect on the base <Project> repo to only test commits where the file <Project>SubRepoVersion.txt was updated. You can do this with a git bisect run <script> where your <script> just skips over commits that don’t have the <Project>SubRepoVersion.txt file updated. For each of the examined commits, <script> would use:

$ gitdist --dist-not-repos=. --dist-version-file=<Project>SubRepoVersion.txt checkout %%version%%

before doing the configure, build, and test. (Here, %%version%% is replaced with the SHA1 for each repo listed in the <Project>SubRepoVersion.txt file. The --dist-version-file argument is already supported by TriBITS [0]; I just want to change the name of the SHA1 place-holder from _VERSION_ to %%version%% to avoid bad matches. (see #135 ))

The outcome of this multi-repo git bisect would produce a consecutive pair of <Project>SubRepoVersion.txt files that would bound the range of commits in each of the repos where the defect was first added.

When git bisect stopped (at a detached head state for all the repos), the developer could view the bounded range of commits for each repo where the defect was added with:

$ gitdist --dist-committed-version-file=<Project>SubRepoVersion.txt \
    --dist-version2=-1 log --name-status HEAD ^%%version2%%

where %%version2%% would be replaced with the SHA1 for the repo listed in the committed <Project>SubRepoVersion.txt file from the last commit (specified by the --dist-version2=-1 argument) where that file was updated [1].

From there, the developer could manually examine and perhaps further bisect the commits in the different repos to try to find the breaking commit(s). In general, since you can’t just try any combination of commits in the different repos, it would be up to the developer to figure out strategies for looking for the breaking commit(s) once they had this reduced range of commits from each repo. However, if just one repo was updated between those two consecutive repo sets, then one could use git bisect on just that one repo and find the exact commit that added the defect (assuming the commits were "complete" and not broken).

---

SIDE NOTE:

The default argument --dist-committed-version-file=<Project>SubRepoVersion.txt could be stored in the version-controlled .gitdist_default.py file [2] so that the developer would just have to type:

$ gitdist --dist-version2=-1 log --name-status HEAD ^%%version2%%

Also, developers could examine any range of commits using:

$ gitdist --dist-version=<sha1-1> --dist-version2=<sha1-2> \
    log HEAD %%version%% ^%%version2%%

where <sha1-1> and <sha1-2> are SHA1s for the base VERA repo version that have updated <Project>SubRepoVersion.txt files.

The gitdist script could also support the single arg --dist-version-pair=<sha1-1>,<sha1-2> to replace the two args --dist-version=<sha1-1> and --dist-version2=<sha1-2> to allow:

$ gitdist --dist-version-pair=<sha1-1>,<sha1-2> \
    log HEAD %%version%% ^%%version2%%

This would make it very easy to poke around and look at different ranges of commits for compatible sets of repos. I don’t think that git submodules supports that level of flexibility (at least not without some scripting and using git submodule foreach … commands). All that git submodules supports is a git submodule summary command and that only is useful before you commit.

---

The challenge is to commit the <Project>SubRepoVersion.txt file to the base VERA repo in such a way that is transparent to regular developers and avoids silly merge conflicts like you get with git submodules [3]. The idea I had was to piggyback this off of the post-push CI server. So whenever the post-push CI build passed (or at least the build and most of the tests passed, this could be configurable), the CI server would create a new commit in the base repo with the updated <Project>SubRepoVersion.txt file and then push it to the main development branch in the base project repo. That way, you would have known good versions of sets of repos committed that would be safe to bisect on. (In fact, I think that I could add this to TribitsCTestDriverCore.cmake without too much effort so that this could be turned on with a flip of a switch for any TriBITS project.) These special commits could also contain info in the log message about what got tested, if the build passed and how many tests passed. That info could be used with more custom git bisect run <script> scripts in determining what commits to test or not.

The advantage of this post-push CI implementation is that developers could just continue to do gitdist pull, create git commits, and gitdist push like they have always done and just forget about this <Project>SubRepoVersion.txt file in general. But this file would be there in case they wanted to use git bisect or just checkout older versions later as described above. (That is very different than git submodules that forces every developer and user to know about and how to use git submodules and avoids the gotchas of using git submoudles [3].)

One potential disadvantage of the post-push CI implementation is that is that every CI iteration would create a new commit in the base repo that just contained the updated <Project>RepoVersion.txt file. Some people might see these extra commits as clutter. Personally, I think I would like to see these commits since they make the compatible sets of commits explicit and easy to understand. And note that gitsubmodules also creates new commits in the base repo whenever the state of the submodule repos are saved. So this would be no worse than when using git submodules.

Another approach I worked out on paper was to commit the <Project>SubRepoVersion.txt file as part of pre-push CI testing and pushing which would be automated with the checkin-test.py script. But that would be a much more complex process and it would have to deal with merge conflicts of all types. The advantages of this approach over the post-push CI approach described above is that it would create the cleanest git history (because it could add the updated <Project>SubRepoVersion.txt file to the top commit when it amended it and therefore would not create any new commits) and it would not require the setup and usage of a post-push CI server. The disadvantage is that it would come at some extra complexity both in the checkin-test.py script and for average developers if there were merge conflicts in any of the repos. Therefore, I am not sure it is worthwhile to implement this pre-push CI approach. I suspect that the post-push CI approach will be plenty sufficient and it is much easier to implement and will be more robust (it will never have to deal with a merge conflict if only the post-push CI server is updating the <Project>SubRepoVersion.txt file).

References:

• [0] https://tribits.org/doc/TribitsDevelopersGuide.html#gitdist-dist-help-repo-versions
• [1] https://docs.google.com/document/d/1WKs8rJhI3037yKGnEVMhIx9dPN7a7uFRM5isdNAhAXI/edit#bookmark=id.6vz2rx12wf4e
• [2] .gitdist.default.py: https://docs.google.com/document/d/1WKs8rJhI3037yKGnEVMhIx9dPN7a7uFRM5isdNAhAXI/edit#bookmark=id.db16nyaeyrm1
• [3] “Merging Submodule Changes” https://git-scm.com/book/en/v2/Git-Tools-Submodules

[bartlettroscoe]

FYI: This approach was implemented for SPARC on top of gitdist a long time ago. Still, it would be nice to have built-in support for this.