rebis-dev: Why 3402 commits ahead of, 3394 commits behind master?

[triska]

When I visit https://github.com/mthom/scryer-prolog/tree/rebis-dev, github shows:

This branch is 3402 commits ahead of, 3396 commits behind master.

Why such unexpectedly high numbers? The two branches share almost all commits, I expect rebis-dev to contain almost all commits from master, with possibly a few rare exceptions of recent commits that are no longer relevant for rebis-dev. Note that master itself has only 3884 commits, and rebis-dev has 3890 commits at the moment.

[mthom]

I created the abolish_2 branch nearly a year ago to begin scrubbing Term from the codebase, then rebased the pstr_in_heap branch on top of it.. at some point a discrepancy in the commit history was introduced.

[triska]

Still, it seems that the branches should/could have many more commits in common. Maybe this can be reflected somehow to make it clearer what the branches actually have in common. Maybe you @Skgland know how to do this? I would greatly appreciate if you could look into this.

[Skgland]

Oh no, this looks bad.

Based on a a visualized commit graph master and rebis-dev diverged after 80def45d, which was committed 2018-09-28, and I can't see a merge between those branches either way since.

The main problem I see are all the things from master that appear to have been rebased onto rebis-dev that are complicating the history due to them necessarily getting a new commit id in the process. A rebase of rebis-dev onto master or a merge with master wouldn't have had this problem. I don't know a way to fix this automatically or easily.

The only thing I can come up with that might work, but which will definitely be tedious, would be to

1. checkout the most recent commit after the divergence that is just before the first rebased commit
2. merge the original commits from master rather than rebasing them
   • ensuring the merged result is identical to the rebased result
3. rebase the remaining rebis-dev history onto this newly created merge commit
4. repeat until done

[Skgland]

Not that it helps, but maybe abolish_2 was created from an already diverged branch rather than from master?

[bakaq]

Can't we just make a new branch from master and then cherrypick the commits that matter from the current rebis-dev? That would also involve a bunch of merge conflicts, but I think it would be way less painful than the whole rebase dance.

[triska]

Thank you for all suggestions and comments! Almost everything would be better than the situation we have now, where the branches are presented almost as if they don't have anything in common. We have many issues and discussions that link to commits in master, and it seems worthwhile to keep as many of them working as possible. Even if we can only preserve the existing commits in master up to somewhere in 2023, it would already be a huge improvement.

P.S.: Especially since in 2023, we already had a rebis-dev branch that closely followed master, maybe this could serve as the basis.

[triska]

The following may be a way out: We can simply add all rebis-dev changes in a single commit on top of master, by performing the following sequence of commands which I post here and explain in the following:

$ git fetch origin
$ git checkout master && git commit -a
$ git diff origin/master origin/rebis-dev | git apply
$ git add src/functor_macro.rs
$ git commit -a
$ git push origin master:rebis-dev --force
$ git reset --hard HEAD^

Explanation

First, make sure everything is up to date, master is checked out and all local changes are committed to the repository:

$ git fetch origin
$ git checkout master && git commit -a

Then, apply all changes of rebis-dev verbatim to master, i.e., do not attempt to apply the commits (which do not apply cleanly) but the textual changes to files:

$ git diff origin/master origin/rebis-dev | git apply

At this point, I get the following messages:

src/rebis.patch:19011: trailing whitespace.
                    (HeapCellValueTag::Cons | HeapCellValueTag::Fixnum | // HeapCellValueTag::Char | 
src/rebis.patch:22079: space before tab in indent.
                       if h == focus {
src/rebis.patch:22080: space before tab in indent.
                           break;
src/rebis.patch:22081: space before tab in indent.
                       } else {
src/rebis.patch:22082: space before tab in indent.
                           focus = h;
warning: squelched 13 whitespace errors
warning: 18 lines add whitespace errors.

I have ignored these messages, they do not influence the end result.

At this point, all changes of rebis-dev are applied to the repository, but not yet committed.

To these changes, add the file src/functor_macro.rs, which is new in rebis-dev and now also available in the repository:

$ git add src/functor_macro.rs

Now, commit all these changes with:

$ git commit -a

This yields a single big commit (69 files changed, ca. 10_000 insertions and deletions) on top of master.

The commit will look like this: https://github.com/triska/scryer-prolog/commit/bd919ddaade058317a044a8bd04852582e6c449b

The result (i.e., master with this commit applied) is exactly identical to the remote rebis-dev branch.

Verification:

$ git diff origin/rebis-dev
$ (i.e., empty diff)

At last, push master to the remote rebis-dev branch:

$ git push origin master:rebis-dev --force

And then reset master to its original state by undoing the topmost commit which contains the rebis-dev changes:

$ git reset --hard HEAD^

I think the result is acceptable, much better than destroying the existing commit history of several years, especially since commit IDs are also used in commit messages and discussions that refer to them.

[Skgland]

The following may be a way out: We can simply add all rebis-dev changes in a single commit on top of master, by performing the following sequence of commands which I post here and explain in the following:

$ git fetch origin $ git checkout master && git commit -a $ git diff origin/master origin/rebis-dev | git apply $ git add src/functor_macro.rs $ git commit -a $ git push origin master:rebis-dev --force $ git reset --hard HEAD^

Explanation

First, make sure everything is up to date, master is checked out and all local changes are committed to the repository:

$ git fetch origin $ git checkout master && git commit -a Then, apply all changes of rebis-dev verbatim to master, i.e., do not attempt to apply the commits (which do not apply cleanly) but the textual changes to files:

$ git diff origin/master origin/rebis-dev | git apply At this point, I get the following messages:

src/rebis.patch:19011: trailing whitespace.
                    (HeapCellValueTag::Cons | HeapCellValueTag::Fixnum | // HeapCellValueTag::Char | 
src/rebis.patch:22079: space before tab in indent.
                       if h == focus {
src/rebis.patch:22080: space before tab in indent.
                           break;
src/rebis.patch:22081: space before tab in indent.
                       } else {
src/rebis.patch:22082: space before tab in indent.
                           focus = h;
warning: squelched 13 whitespace errors
warning: 18 lines add whitespace errors.

I have ignored these messages, they do not influence the end result.

At this point, all changes of rebis-dev are applied to the repository, but not yet committed.

To these changes, add the file src/functor_macro.rs, which is new in rebis-dev and now also available in the repository:

$ git add src/functor_macro.rs

Now, commit all these changes with:

$ git commit -a This yields a single big commit (69 files changed, ca. 10_000 insertions and deletions) on top of master.

The commit will look like this: triska@bd919dd

The result (i.e., master with this commit applied) is exactly identical to the remote rebis-dev branch.

Verification:

$ git diff origin/rebis-dev $ (i.e., empty diff) At last, push master to the remote rebis-dev branch:

$ git push origin master:rebis-dev --force

And then reset master to its original state by undoing the topmost commit which contains the rebis-dev changes:

$ git reset --hard HEAD^

I think the result is acceptable, much better than destroying the existing commit history of several years, especially since commit IDs are also used in commit messages and discussions that refer to them.

This generates the diff between master and rebis-dev, then applies it to master, this obviously ends in the state of rebis-dev. But it also completely ignores/forcefully undoes all changes in master that are not in rebis-dev.

[triska]

But it also completely ignores/forcefully undoes all changes in master that are not in rebis-dev.

To clarify: After these steps, master is unchanged! The only purpose of these changes is to more closely align the history of rebis-dev with that of the existing master branch. At the end of these steps, only the history of rebis-dev is changed, and the content of rebis-dev is exactly the same as now!

At the end, we are conceptually in exactly the same position as we are now: There are new changes in rebis-dev (now all in a single commit), and master is the same as now.

Any changes that happened in the last few weeks must be integrated when (eventually) merging rebis-dev, exactly the same as now, such as a change to the README that took place a few days ago and is currently not yet in rebis-dev.

[notoria]

The master that was merged into rebis-dev was 3fa6b69a. The master and rebis-dev can be visualized with git log --topo-order --oneline --graph origin/master origin/rebis-dev.

For the cherry-pick method, the commits are 15cbb92a, e155acb1, 31e7f210, a82eec92. These commits can be on a new branch without conflict:

$ git checkout 8ac663d7
$ git checkout -b rebis-next
$ git cherry-pick 15cbb92a e155acb1 31e7f210 a82eec92

The new branch will start after #2442 .

[triska]

@notoria, that's brilliant, thank you a lot!

I tried this method and pushed the result to https://github.com/triska/scryer-prolog/tree/rebis-next .

This branch looks much better in the sense that the number of commit differences is now much smaller:

The 4 commits are exactly the key commits we want from rebis-dev.

Even so, the resulting branch apparently cannot be automatically merged into master. I tried creating a PR and get:

[notoria]

Now the new branch needs to be rebased on master. It could also be done step by step like a rebase at each merge (git rebase eaa95293 to rebase on #2439 and so on). This is the part where conflicts need to be resolved.

[mthom]

I've done as @notoria suggested and pushed a rebis-next branch.

[triska]

Thank you a lot Mark for this awesome work!

It now remains to ensure that rebis-next contains all changes from rebis-dev, and also all changes that have since been applied to master. When I do:

$ git diff origin/rebis-dev origin/rebis-next

I see that rebis-next contains changes that are neither in rebis-dev nor in master, such as the renaming of REPLCodePtr to ReplCodePtr, as well as removal of several fields and functions which were apparently not needed in both branches. I suppose these are intended changes that are now also incorporated.

Here is the diff output:

https://www.metalevel.at/prolog/scryer/rebis-differences.txt

I would greatly appreciate any feedback and experience reports with rebis-next, so that rebis-next can soon replace rebis-dev! Thank you a lot!

[triska]

rebis-next looks good, thank you a lot! I suggest (force-)pushing this to rebis-dev and deleting rebis-next so that we can resume work on rebis-dev which several issues now already reference.

[triska]

This is now vastly improved, thank you a lot!