(Re-)Initialization of a cloned repo fails

[pets-er]

Hi there! Nice work on this tool.

I had a problem twice when trying to (re-)initialize the same cloned repo. This was not a problem on my laptops. One of the failed times was on a server that is not managed by me (work), using ssh, one of those times on termux, also using ssh.

The error that I get is this: .git/crypt/transcrypt: eval: line 192: syntax error near unexpected token `('

It should also be noted that this leaves the repo in an encrypted state, but transcrypt thinks it is properly initialized.

Do you have an idea how I could fix/diagnose this? As far as I know, the systems run bash (I checked using ps on the server) - the requirements should also be fulfilled. Maybe there is something odd about the setup or some implicit assumption about what the system should be capable of that is not fulfilled. The line 192 does not seem suspicious to me - it is a comment line without a (, maybe something happening in the function above it?

Thank you!

[jmurty]

Hi, thanks.

Which version of transcrypt are you using? And which version of bash is on the systems that show the problem (run /usr/bin/env bash --version)?

If it's the latest version from the main branch in this repo the problem line would be this one: https://github.com/elasticdog/transcrypt/blob/efd42e5439aa27c2c21c19de1aaab134ad883c28/transcrypt#L192

Looking at that line, perhaps there is some content in your repository's file listing that is not properly escaped or handled when piped into the eval portion of the loop when run with the version of bash running on those systems.

Debugging bash scripts is a pain, and the code in the _list_encrypted_files() function contains some particularly tricky black magic to try and handle file names with unusual/problematic characters.

I would recommend starting by tweaking the line slightly to see if you can get it to work on the problem systems. Try these two slight tweaks, which might plausibly help:

• while read -r file; do eval "echo ${file}"; done
• while read -r file; do echo "${file}"; done
• while read -r file; do eval 'echo $file'; done

Let me know how you go.

[jmurty]

I should mention that – assuming the line I linked above is indeed the problem line – you can run just the problematic function for easier testing with the --list param, e.g. .git/crypt/transcrypt --list

[pets-er]

Hi! First off: GNU bash, version 5.1.8(1)-release (x86_64-redhat-linux-gnu)

My trials were always fairly fresh transcrypt pulls.

I will let you know that I use this for my obsidian vault and have all .md files encrypted. There are natural writing file names and special characters - but only what obsidian allows. The strange thing is that I can re-clone my repo and initialize it on my arch based system using the current AUR-packaged version (2.3.1-1), so it has to be a combination.

Just checked with the list command, it fails just before a file with a round bracket in the name, but I am not entirely sure if this is what is causing it. Just renaming the file with a bracket in the name did not immediately fix it, but maybe this has to be done properly in git etc.

[jmurty]

Ah okay, it definitely seems like you're hitting a transcrypt bug handling one of your file names.

Here is a command you can run which is a trimmed down version of most of what the _list_encrypted_files() function contains, leaving off the final problem line with the while loop. You can run this to bypass transcrypt entirely and debug just the commands it is running.

IFS=$'\n' git -c core.quotePath=false ls-files -z | tr '\0' '\n' | git -c core.quotePath=false check-attr filter --stdin 2>/dev/null | { grep ": filter: crypt$" || true } | sed "s|: filter: crypt||"

The command lists all the files in the git repo with metadata printed after the file name, then filters down to those with the metadata that indicates a transcrypt-encrypted file. You can get a sense of what is going on if you run parts of this command, building up the full thing one piped section at a time.

Does piping the above into the different variants of the while command I mentioned above, plus the current while read -r file; do eval "echo $file"; done line, reveal anything useful?

If you can share an example of the problem filename that might let me reproduce the problem locally, though I suspect it may be tricky for me to reproduce since the bug occurs for only some of your systems.

[jmurty]

The latest main version of transcrypt has changes to the _list_encrypted_files() function to improve performance in large repos (#193). These changes aren't yet in an official release and are not in transcrypt version 2.3.1.

So you could probably sidestep this whole issue by using released version 2.3.1 instead of the latest unstable version. But it would be great to figure out and fix this bug, so it doesn't end up in a future release.

[jmurty]

I have been able to reproduce the issue by creating and staging a file with this name: "Terrible file""")))(((][][].secret

Using a different variant of the while loop line does fix the crash, but this has revealed a deeper problem where the first few portions of the overall command run by transcrypt strip out special characters from file names when they are not intended to.

Running this IFS=$'\n' git -c core.quotePath=false ls-files -z | tr '\0' '\n' | git -c core.quotePath=false check-attr filter --stdin 2>/dev/null in a repo containing my "terrible" file prints the file named as just Terrible file which is definitely not right, and which means transcrypt isn't recognising encrypted files when the filename contains some certain characters.

I may need to unwind some of the changes from #193 to ensure filenames with unusual characters are handled correctly, but without causing bad performance for repos with many files.

[jmurty]

Hi @pets-er could you try a potential fix I've applied to transcrypt in PR #205?

[pets-er]

Okay, I checked out the issue-204 branch and initialized. This works! Previously, I also tried the old release version, that also worked.

I have not yet tried on termux, but as the issue seemed to be the unstable version, this seems fine :)

Thank you very much! Is there any more info that you might need?

[jmurty]

Great! Glad to hear the 204 fix worked for you.

Could you list example file name(s) that caused this problem for you? I would like to collect another example or two to add to unit tests I'll write to exercise transcrypt's handling of difficult file names.

Thanks for the report and feedback.

[pets-er]

Yeah so I am not quite sure which ones they exactly are, since I do have a bunch of files. Can you tell me again which would be the quickest way to find them? One of the commands above did not quite work (another did), but I also did not yet have time to play around with.