Improve README

[Riduidel]

Currently, README seems quite clear ... for people knowing the nuts and bolts of git. But for not-so-good developers (like me), things seems a little ... opaque.

Can you add an example of how subtrac transforms a repository (typically, I would like to see how the submodules are transformed into valid commits)? I would also like to see how to add new submodules to a project where subtrac is already used.

[wzel]

I think, some graphical explanation would clarify the merits of git-subtrac.

[apenwarr]

It's notoriously difficult for someone who really understands the guts of a system to write clear documentation for people who don't. If someone would like to try updating the README in a pull request, I'd be happy to accept.

Otherwise I can make some attempt in the next few days, but it won't be perfect :)

[Riduidel]

ok, but first, let's try to see if I understand correctly.

Suppose I have a repository (let's call it myProject) with one branch (let's say master) and in that branch a submodule called mod.

On the initial run of subtrac

• a branch called master.trac will be created
• in that branch, a commit will be created having for parents both the commit adding the submodule in masterAND the commit in the mod project that created the data.

Is it ok ?

And then, how will updates happen ? Should I go into the master.trac branch or is there any other operation that allow me (from the master branch) to update the submodule ?

[Riduidel]

On a technical side, I guess I'll be able to draw that using https://gitgraphjs.com

[NicolaiRuckel]

Something that also should be explained is how to make changes in a subrepo and push those changes to the parent repository with or without pushing the changes in the subrepo to upstream.

[Riduidel]

ok, seems like my understanding of subtrac is not good enough to even try to document it. Let me try to see if I understand correctly ...

The problem subtrac tries to solve is that submodules are

1. Scattered in repository
2. Have independent history from the repository

So what subtrac do is rewriting submodules as commits with multiple parents : one is the classical commit parent, and the other(s?) is the commit of the submodule source. But this is done is the famous .trac branch (due to reasons I don't yet understand).

However, I have at least understood that, if I want to add a submodule to my project, I do as usual then call git subtrac update to have that new submodule transformed into a commit into the .trac branch. I guess that the there is also a link between the commit in the .trac branch and the commit in the standard branch, to have the link correctly memorized.

Well, I'm gonna test all that to make sure I understand things correctly.

[apenwarr]

Thanks for volunteering to help. Let me try to answer your questions in bulk. Not sure whether this is the ideal medium or not for a complex discussion, but we can give it a shot!

I'll explain using git techie talk for now. Let me know if there are any parts that you don't understand. Hopefully you'll have some luck paraphrasing it into something more comprehensible to normal people :)

Can you add an example of how subtrac transforms a repository (typically, I would like to see how the submodules are transformed into valid commits)? I would also like to see how to add new submodules to a project where subtrac is already used.

So, the first thing to note is that subtrac doesn't actually transform anything. There are separate DAGs of commits (which each point at trees, which each point at trees and blobs and maybe submodule commits). The DAGs are:

• the parent project (aka supermodule). This is the toplevel one that refers (somewhere in its tree) to submodule commits. If there are recursive submodules (ie. submodules that point to further submodules), then a submodule can also be a supermodule. A supermodule can point, at different points in its history, to different points in the history of its submodules.

• each submodule also has its own completely separate history (DAG of commits).

Normally, these separate DAGs are stored in separate git repositories: one per sub/super module. But there is nothing stopping you from fetching one git project's history into another; you can do it at any time with just 'git fetch https://...whatever... master' for example, and then FETCH_HEAD will refer to the history of a whole different module.

What git-subtrac does is construct a new DAG (the trac branch), with new commit objects. Those new commit objects refer to the submodule commit objects that were referred to by supermodule trees. In the trac branch, instead of referring to the submodule commits via trees, we refer to them via commit parents.

In a sense, the trac branch is created by "transforming" the supermodule commit DAG. However, it's important to note that:

• the trac branch does not contain any of the supermodule's commits, trees, or blobs. If you push a copy of the trac branch somewhere, it links (directly or indirectly) to all your submodules, but not to any of the code in the supermodule.

• thus, the original supermodule DAG is still needed if you want the supermodule's code. This is different from git-subtree, which does a similar sort of transformation, but you merge the resulting new DAG into your supermodule's original DAG as if it were a branch.

• thus, the original supermodule DAG and all the submodule DAGs still exist and are still needed. We just have one more DAG, and it happens to also refer to all the submodule DAGs.

Suppose I have a repository (let's call it myProject) with one branch (let's say master) and in that branch a submodule called mod.

On the initial run of subtrac

• a branch called master.trac will be created in that branch, a commit will be created having for parents both the commit adding the submodule in masterAND the commit in the mod project that created the data.

Almost: the new commit does not include the commit that adds the submodule. It only includes the submodule's commit.

• And then, how will updates happen ? Should I go into the master.trac branch or is there any other operation that allow me (from the master branch) to update the submodule ?

Whenever you update the submodule links from the supermodule (in the usual way for submodules: git commit in the submodule, then git add + commit in the supermodule), you need to run 'git subtrac update'. This will extend the trac branch to include any new submodule commits references by your new supermodule commit, if there are any.

If you make a commit to the supermodule that doesn't change any submodule links, the trac branch will be unchanged, because of the way git-subtrac prunes redundant commits.

If instead of adding one commit to the submodule, you rebase or revert or reset or switch branches or whatever, exactly the same thing happens: exactly one new commit is added to the trac branch that references the new top-level commit in the submodule branch, referred to by the one new commit in the supermodule branch.

Note that you can look at the trac branch in something like gitk and it... sort of makes sense. It gets a little tangled :)

On a technical side, I guess I'll be able to draw that using https://gitgraphjs.com

Cool, I've never seen that before! I don't know if it'll work with a README.md (will it?) but maybe it would work with readthedocs.io or something. Or just a blog post somewhere that we can link to.

Something that also should be explained is how to make changes in a subrepo and push those changes to the parent repository with or without pushing the changes in the subrepo to upstream.

Yes, this is a very important feature. Note that the trac branch contains the complete history of the superproject's references to its submodules, no matter how many rebases or branch changes or resets they might have gone through.

You can maybe think of it like 'git reflog' - the submodule history doesn't just "fast forward" like with git-subtree or git merge. You can roll it backwards, sideways, etc, and git-subtrac remembers each step along the way (or rather, each step where you snapshotted it with a commit in the supermodule).

Another really important note is that branches (or tags, refs, etc) don't matter at all in the submodules anymore. The supermodule refers to commit ids in the submodules, not branches. When you branch the supermodule, you also (effectively) branch the submodules at the same time, and you can do whatever you want in the submodules without affecting any other branches. The only catch is that you need a .trac branch for every branch in the supermodule.

(When you merge your supermodule branch back into master or whatever, the .trac branch will reflect the same merge. 'git subtrac update' will regenerate the .trac branch from scratch, but the result will be a branch that looks like a merge of the master.trac and feature.trac branches.)

As a result, forks and branches work exactly the same way. Someone can fork your supermodule on github and make their own, and they can edit the submodules as much as they want as part of the fork, and you can merge the fork back in later (as long as you also git subtrac update after the merge).

Now, as for the "upstream" copy of the submodule, that's just a different version of the submodule's history, with its own branches. At any time, you can cd into the submodule from your supermodule, and do something like git push github.com:... HEAD:master and send the current version upstream. But this isn't necessary; it doesn't affect anything about the superproject. It's just another copy of the repo.

The problem subtrac tries to solve is that submodules are

1. Scattered in repository
2. Have independent history from the repository

So what subtrac do is rewriting submodules as commits with multiple parents: one is the classical commit parent, and the other(s?) is the commit of the submodule source. But this is done is the famous .trac branch (due to reasons I don't yet understand).

(I'm answering all the questions in sequence in this mega-comment; this is a duplicate but I'll answer it here anyway :)) Almost true, except as mentioned above, the .trac branch does not use the superproject commit as a parent. It only uses the submodule commits as parents, as well as other synthetic trac commits if needed.

However, I have at least understood that, if I want to add a submodule to my project, I do as usual then call git subtrac update to have that new submodule transformed into a commit into the .trac branch.

Not exactly transformed: augmented. We build a new DAG of commits that includes the entire, original, DAG of commits from the submodule history.

I guess that the there is also a link between the commit in the .trac branch and the commit in the standard branch, to have the link correctly memorized.

Actually no, there is no link. git submodule links are links from a tree to a commit. That is, a "directory" of your supermodule is a commit in a submodule (which contains its own files, so it appears as a directory in your worktree). A normal submodule link therefore is not referenced from the supermodule commit directly; only indirectly through the commit's tree or one of its subtrees.

The .trac branch is a bunch of synthetic objects, but none of those objects are needed in order to checkout the supermodule or any of its submodules. It's purely a bookkeeping mechanism, telling git how to push or pull all the submodule commits to wherever you want to push or pull them, in a nice bundle. But the trac DAG is never referred to by anything in the supermodule DAG (which is unmodified), nor vice versa. You need them both.

(It's tempting to think about merging the trac branch into the supermodule DAG, so then you only have to push and pull a single branch! But that would clutter your project history horribly, and everyone hates that. You'd see every single commit from every single submodule whenever you 'git log' in your supermodule. Bleh.)

Phew. Hope this helps! If you can make that more readable, I'll be very impressed and thankful :)

[Riduidel]

I've started a weird webpage (to be used in gh-pages). Currently, I have a small example of the submodule problem. You can take a look there http://nicolas-delsaux.hd.free.fr/subtrac/ What do you think ?