Merge and Rebase with remote repo fail

[chikathreesix]

When a encrypted file is changed in a remote repo, I am facing issues with merge and rebase.

Merge

With merge, the issue happens when there is a conflict in a secret file between my repo and a remote repo. The merge itself succeeds with showing there is a conflict in the secret file. But I can't see the diffs in the file. The lines like "<<<<<<< HEAD" are supposed to be in the file.

I did this with commands below. In this situation, I am in a branch "feature/a" with changes in a secret file. And "origin/master" also has a change in the secret file

$ git fetch
$ git pull origin master

If there is a change in a secret file in the remote repo but no conflict with my local repo, it succeeds :)

Rebase

Whether there is a conflict or not, if there is a change in a secret file in a remote repo, rebase always fails as follows.

This is a same situation as the merge above. I did this with commands below.

$ git fetch
$ git rebase origin/master

The rebase seems to succeed but I am getting messages below.

File not encrypted
error: external filter "git-crypt" smudge "key name here" failed 1
error: external filter "git-crypt" smudge "key name here" failed

This message is shown for each secret file that were changed. Then if I do "git status", it shows the secret file is modified. But once I got the messages above, I can't do "git stash" or "git reset". The secret file keeps failing to encrypt and there is no way to turn the file back to encrypted one without just remove the file from the repo and add this again.

Thanks

[AGWA]

Hi,

Thanks for the report.

Conflict resolution is unfortunately something which git-crypt does not handle at the moment. Fortunately I've got some ideas for how to fix it. I'll do some experimentation, and if my ideas work, I should be able to push a prototype fix in the next few days.

I'm unable to reproduce the issue with the rebase. As long as there's no conflict, it should just work. What version of Git are you using?

[chikathreesix]

Thanks for your response :) Hope there is a way to fix it.

I'm using "git version 1.8.5.2 (Apple Git-48)".

[AGWA]

Unfortunately my ideas did not work. The problem is that Git applies merges on the clean (i.e. encrypted) versions of files, not the checked-out and smudged (i.e. decrypted) versions, so if an encrypted file is modified on more than one branch being merged, it's treated like a binary merge and fails, without even adding conflict markers. One solution may be to write a git-crypt merge driver, but that would be a long-term project.

Even with Git 1.8.5.2 I'm unable to reproduce an issue with rebasing onto a branch containing secret file changes. Was the secret file also modified in the branch you were rebasing (i.e. the local repo)?

Out of curiosity, what are you using git-crypt for? Lately I've been getting the impression that git-crypt is being used for a bit more than I originally intended, which is why I've never seriously looked at merges before. I'm perfectly fine with expanded use of git-crypt; I just need to know what additional challenges I'm facing ;-).

[chikathreesix]

Thank you for trying that.

Rebasing fails whether a local repo has secret file changes or not. And the problem is, once I failed to encrypt, it keeps showing an error and I can not do anything to the secret file. Is there any way of rolling the file back at least?

By the way, I use git-crypt for encrypting configuration files for rails application that have database password, database user name and so on.

[AGWA]

@chikathreesix wrote:

Rebasing fails whether a local repo has secret file changes or not.

Is the .gitattributes file the same in both repos? I'm wondering if perhaps the file was never encrypted to begin with due to missing or non-matching .gitattributes in the local repo.

And the problem is, once I failed to encrypt, it keeps showing an error and I can not do anything to the secret file. Is there any way of rolling the file back at least?

I would have thought git-reset, but you tried that. It's hard to say for sure without knowing what state the repo was in. I'll try to do more testing to get git-crypt into this state so I can figure out the best way to get out of it. Also, it's on my short-term TODO to provide a tool to "reset" a git-crypted repo to a sane state.

By the way, I use git-crypt for encrypting configuration files for rails application that have database password, database user name and so on.

Alright, that's exactly the use case I had in mind. I would assume that files like this aren't modified too often, especially in parallel on two branches, so the lack of merge support isn't too much of a deal-breaker. (Of course, rebasing when there have been no changes is a separate issue and ought to work). When I refresh the documentation for the upcoming release, I'll make it clear that git-crypt doesn't support merging yet.

[chikathreesix]

Sorry for my late reply.

Is the .gitattributes file the same in both repos? I'm wondering if perhaps the file was never encrypted to begin with due to missing or non-matching .gitattributes in the local repo.

Yes, both repos have same .gitattributes file.

I would have thought git-reset, but you tried that. It's hard to say for sure without knowing what state the repo was in.

I am not sure how to reproduce this issue. I've tried with a simple repo but couldn't reproduce this. If I could find the cause, I will let you know.

Also, it's on my short-term TODO to provide a tool to "reset" a git-crypted repo to a sane state.

That would be great! thanks.

When I refresh the documentation for the upcoming release, I'll make it clear that git-crypt doesn't support merging yet.

Yeah, I think that would be better.

[jtwang83]

Shouldn't this setting in the .gitattributes file fix this?

[merge] renormalize = true

[cogpie]

@jtwang83 @tadas-s and I (well mainly @tadas-s :) tried this setting but it didn't seem to solve the merging issue.

Thankfully we may not need to alter password files often so our use-case aligns with the original design, however, we can imagine people editing the file before pulling the latest version and then having merge issues (as you always will with any edit to an encrypted binary file, under good encryption :) so this means even changing different lines will result in a merge commit.

@AGWA was there any work on a merge driver? Or even a post-merge commit hook to go back and look at those encrypted files that were conflicted and then take the unencrypted parents and produce a merged file showing any actual conflicts git-style.

[smemsh]

I was also under the impression that renormalize = true (as mentioned by @jtwang83) would work. It does a smudge/clean into a temporary file before doing the merge compares. It was made for the case of differing CR/LFs, they get put into "normalized" form beforehand. So, I think it should work; I'm surprised that it doesn't.

Just to clarify, the mentioned stanza goes into gitconfig, not gitattributes.

[bender-the-greatest]

+1 for this, merging anything that is encrypted is a huge pain with git-crypt

what i don't fully understand is why i can run git diff and see a full diff in the conflicted files in the middle of the conflicted merge, but the conflict markers don't show up in the conflicted file itself. seems like one could decrypt into a temporary file at least and merge that, then re-encrypt the file once the merge is done.

[romanchyla]

Here is a workaround (for people who search for this issue):

git fetch <your-master-branch>
git checkout --ours -- <conflicting-encrypted-file>
git diff origin/master -- <conflicting-encrypted-file>

this will help you 1) restore the original file; then 2) see the differences - you will have to make manual edits

[ryu-jin]

Thank you for git-crypt, it's an amazing project. Is there any plan to fix the merge issue? Cheers