ropensci
Better search for default SSH keys on Windows
This is a follow-up on #329
Currently, the git2r default is to look for SSH keys (id_rsa) in file.path(Sys.getenv("USERPROFILE"), ".ssh"). It seems that this is not always the default location. Can the search be improved?
From my (cursory) look through the git2r source, I can't tell how the putative location of SSH keys is even determined. Is that happening in source here in git2r or even in libgit2? Or is this perhaps inherited from some other dependency?
I have experienced this problem personally in the past past couple of days on my Windows VM. And, indeed, using git2r::cred_ssh_key() to explicitly create a credential, then passing that into relevant git2r calls works.
I have found that the defaults are "good enough", e.g. I have success with indirect use in usethis like so:
use_github(credentials = git2r::cred_ssh_key())
That is due to the relatively recent addition of code that honors where .ssh/ usually lives on Windows ():
https://github.com/ropensci/git2r/blob/358fcd2e8ef5d754d51f456f0b059be79ac119be/R/credential.R#L176-L210
More verbose and, in my case equivalent, is to provide explicit paths:
cred <- git2r::cred_ssh_key(
publickey = fs::path_home(".ssh/id_rsa.pub"),
privatekey = fs::path_home(".ssh/id_rsa")
)
use_github(credentials = cred)
I am tempted to use credentials = git2r::cred_ssh_key() in usethis, when on Windows and when user has not provided an explicit credential. But this seems a bit aggressive, i.e. too much interfering in the natural order of things. Then I contemplate doing this only after an initial failure of git2r/libgit2 to find SSH keys automagically.
Do you understand @stewid (or @jeroen) where the putative location of SSH keys is truly determined and the most proper place to intervene and get better default behavior on Windows?
One thing that makes this ambiguous is that R on Windows uses the user's Documents folder as the HOME directory ~. This is different from most programs that use C:\Users\$username by default.
R sets the environment variable $HOME and therefore C libraries may pick up on this. I haven't tested yet but it looks like libgit2 searches the $HOME location first for config files.
However I am not sure this is applies to the ~/.ssh files because ssh is handled by libssh2.
Also note that ssh keys may also be named id_ed25519, id_ecdsa, etc, see also the ssh vignette.
I'm aware of this ambiguity re: home directory on Windows:
http://happygitwithr.com/ssh-keys.html#git2r-or-some-other-tool-cant-find-ssh-keys-on-windows
I'm beginning to think the least of all evils is to strongly recommend that Windows users create a symbolic link from C:\Users\username\.ssh to C:\Users\username\Documents\.ssh. Or vice versa if they've got it the other way 'round.
Actually interfering in whatever process git2r is (indirectly) using seems a like a good way to create even more headaches and maintenance woes.
If I pursue 👆, I would create some sort of doctor or helper function in usethis to help user achieve this. usethis already depends on fs, so I can provide code (or execute it, even) to make the link.
Windows doesn't really have symlinks like we have on unix. This is one of the major problems porting software to Windows.
There is something on Windows called symlinks but as far as I know these are difficult to create and only available to sysadmins so most users may not be able to use it. Windows does have something similar called native links, but I don't think R and libgit2 can read those.
I think the best solution is that git2r checks for both locations for an .ssh directory in R code and then passes this explicitly pass this location to libgit2 C code (I think that is what @stewid suggests).
In the Happy Git page linked to above, we have example code for doing this:
MKLINK /D "C:\Users\username\Documents\.ssh" "C:\Users\username\.ssh"
provided by @ijlyttle. It's also what's recommended here (re: gitconfig, but same idea):
https://www.onwebsecurity.com/configuration/git-on-windows-location-of-global-configuration-file.html
So there's some evidence that it works in the wild.
I think it would be really nice for git2r and usethis to stay out of the business of actively finding ssh and git config files. I'd rather help the user get a robust set up that's likely to satisfy command line git and libgit2.
I will experiment on Windows and report back ...
@jennybc thanks for your work on this.
I would like to refactor the callback git2r_cred_acquire_cb (https://github.com/ropensci/git2r/blob/master/src/git2r_cred.c#L214) to try harder to provide working SSH credentials from default locations. If a connection fails, the callback is called again and we can try the next default key location, similar to the ssh package. I think we can also add support for using git credentials (https://git-scm.com/docs/gitcredentials). I can create a new branch credentials for this.
I just deleted my last comment above and moved it to a new issue: #356 Set up and troubleshooting advice for ssh-agent on Windows
because I think there are at least two distinct issues around getting SSH to "just work" on Windows.
Yes your plan to look harder in default locations before resorting to the ssh-agent sounds fantastic! It would have benefits beyond Windows, if I understand you correctly.
It seem like the relevant parts exist, but right now a lot of the burden still lies on the user, even though the main hiccup (multiple possible locations for .ssh/) is quite predictable.
@jennybc I have made an internal test function in the credentialsbranch that tries to list available ssh keys. Does it find the keys on a Windows machine?
Example how to use it:
.Call(git2r:::git2r_ssh_keys)
#> private: /home/stefan/.ssh/id_ed25519 public: /home/stefan/.ssh/id_ed25519.pub
#> private: /home/stefan/.ssh/id_rsa public: /home/stefan/.ssh/id_rsa.pub
#> NULL
No it does not find my keys on Windows. But they are there.
> .Call(git2r:::git2r_ssh_keys)
NULL
> fs::dir_ls(fs::path_home(), all = TRUE, regexp = "ssh")
C:/Users/JennyVM/.ssh
> fs::dir_ls(fs::path_home(".ssh"))
C:/Users/JennyVM/.ssh/2017-12-windows-private-key.ppk
C:/Users/JennyVM/.ssh/agent.env
C:/Users/JennyVM/.ssh/id_ed25519
C:/Users/JennyVM/.ssh/id_ed25519.pub
C:/Users/JennyVM/.ssh/id_rsa
C:/Users/JennyVM/.ssh/id_rsa.pub
C:/Users/JennyVM/.ssh/known_hosts
I'm not on Windows right now, but I would assume that will look in C:/Users/JennyVM/Documents, no? Which would miss my keys.
I think you are right. I have changed again. Now I'm only trying to find the paths to: "%HOME%\\", "%HOMEDRIVE%%HOMEPATH%\\" and "%USERPROFILE%\\"
Installed from aea8f2f9a3502fe70e5ed310792f2799490c9e5d, I see this:
> .Call(git2r:::git2r_ssh_keys)
path: C
path: C
path: C
NULL
I think that is because I didn't map the wide character string (from ExpandEnvironmentStringsW) to utf8. I have now fixed that (I hope)
The reason I'm trying to do this in C code is that the libgit2 callback (if the remote host requires authentication) provides information about the requested authentication type. So instead of always doing the search before calling e.g. clone, my idea is to find the ssh keys when they are needed.
I can't compile on Windows from 6e191c200f63e9cd0af4a6ee1f00fd6403043a68:
WARNING: this package has a configure script It probably needs manual configuration
** libs rm -f git2r.dll git2r.o git2r_S3.o git2r_arg.o git2r_blame.o git2r_blob.o git2r_branch.o git2r_checkout.o git2r_clone.o git2r_commit.o git2r_config.o git2r_cred.o git2r_diff.o git2r_error.o git2r_graph.o git2r_index.o git2r_libgit2.o git2r_merge.o git2r_note.o git2r_object.o git2r_odb.o git2r_oid.o git2r_push.o git2r_reference.o git2r_reflog.o git2r_remote.o git2r_repository.o git2r_reset.o git2r_revparse.o git2r_revwalk.o git2r_signature.o git2r_stash.o git2r_status.o git2r_tag.o git2r_transfer.o git2r_tree.o "C:/PROGRA~1/R/R-34~1.3/bin/x64/Rscript.exe" "../tools/winlibs.R" 0.27.2 c:/Rtools/mingw_64/bin/gcc -I"C:/PROGRA~1/R/R-34~1.3/include" -DNDEBUG -I../windows/libgit2-0.27.2/include -DR_NO_REMAP -DSTRICT_R_HEADERS -O2 -Wall -std=gnu99 -mtune=generic -UNDEBUG -Wall -pedantic -g -Og -fdiagnostics-color=always -c git2r.c -o git2r.o c:/Rtools/mingw_64/bin/gcc -I"C:/PROGRA~1/R/R-34~1.3/include" -DNDEBUG -I../windows/libgit2-0.27.2/include -DR_NO_REMAP -DSTRICT_R_HEADERS -O2 -Wall -std=gnu99 -mtune=generic -UNDEBUG -Wall -pedantic -g -Og -fdiagnostics-color=always -c git2r_S3.c -o git2r_S3.o c:/Rtools/mingw_64/bin/gcc -I"C:/PROGRA~1/R/R-34~1.3/include" -DNDEBUG -I../windows/libgit2-0.27.2/include -DR_NO_REMAP -DSTRICT_R_HEADERS -O2 -Wall -std=gnu99 -mtune=generic -UNDEBUG -Wall -pedantic -g -Og -fdiagnostics-color=always -c git2r_arg.c -o git2r_arg.o c:/Rtools/mingw_64/bin/gcc -I"C:/PROGRA~1/R/R-34~1.3/include" -DNDEBUG -I../windows/libgit2-0.27.2/include -DR_NO_REMAP -DSTRICT_R_HEADERS -O2 -Wall -std=gnu99 -mtune=generic -UNDEBUG -Wall -pedantic -g -Og -fdiagnostics-color=always -c git2r_blame.c -o git2r_blame.o c:/Rtools/mingw_64/bin/gcc -I"C:/PROGRA~1/R/R-34~1.3/include" -DNDEBUG -I../windows/libgit2-0.27.2/include -DR_NO_REMAP -DSTRICT_R_HEADERS -O2 -Wall -std=gnu99 -mtune=generic -UNDEBUG -Wall -pedantic -g -Og -fdiagnostics-color=always -c git2r_blob.c -o git2r_blob.o c:/Rtools/mingw_64/bin/gcc -I"C:/PROGRA~1/R/R-34~1.3/include" -DNDEBUG -I../windows/libgit2-0.27.2/include -DR_NO_REMAP -DSTRICT_R_HEADERS -O2 -Wall -std=gnu99 -mtune=generic -UNDEBUG -Wall -pedantic -g -Og -fdiagnostics-color=always -c git2r_branch.c -o git2r_branch.o c:/Rtools/mingw_64/bin/gcc -I"C:/PROGRA~1/R/R-34~1.3/include" -DNDEBUG -I../windows/libgit2-0.27.2/include -DR_NO_REMAP -DSTRICT_R_HEADERS -O2 -Wall -std=gnu99 -mtune=generic -UNDEBUG -Wall -pedantic -g -Og -fdiagnostics-color=always -c git2r_checkout.c -o git2r_checkout.o c:/Rtools/mingw_64/bin/gcc -I"C:/PROGRA~1/R/R-34~1.3/include" -DNDEBUG -I../windows/libgit2-0.27.2/include -DR_NO_REMAP -DSTRICT_R_HEADERS -O2 -Wall -std=gnu99 -mtune=generic -UNDEBUG -Wall -pedantic -g -Og -fdiagnostics-color=always -c git2r_clone.c -o git2r_clone.o c:/Rtools/mingw_64/bin/gcc -I"C:/PROGRA~1/R/R-34~1.3/include" -DNDEBUG -I../windows/libgit2-0.27.2/include -DR_NO_REMAP -DSTRICT_R_HEADERS -O2 -Wall -std=gnu99 -mtune=generic -UNDEBUG -Wall -pedantic -g -Og -fdiagnostics-color=always -c git2r_commit.c -o git2r_commit.o c:/Rtools/mingw_64/bin/gcc -I"C:/PROGRA~1/R/R-34~1.3/include" -DNDEBUG -I../windows/libgit2-0.27.2/include -DR_NO_REMAP -DSTRICT_R_HEADERS -O2 -Wall -std=gnu99 -mtune=generic -UNDEBUG -Wall -pedantic -g -Og -fdiagnostics-color=always -c git2r_config.c -o git2r_config.o c:/Rtools/mingw_64/bin/gcc -I"C:/PROGRA~1/R/R-34~1.3/include" -DNDEBUG -I../windows/libgit2-0.27.2/include -DR_NO_REMAP -DSTRICT_R_HEADERS -O2 -Wall -std=gnu99 -mtune=generic -UNDEBUG -Wall -pedantic -g -Og -fdiagnostics-color=always -c git2r_cred.c -o git2r_cred.o git2r_cred.c: In function 'git2r_path_from_environment_variable': git2r_cred.c:233:5: error: 'len_utf8' undeclared (first use in this function) len_utf8 = WideCharToMultiByte( ^ git2r_cred.c:233:5: note: each undeclared identifier is reported only once for each function it appears in git2r_cred.c:234:18: error: 'WC_ERR_INVALID_CHARS' undeclared (first use in this function) CP_UTF8, WC_ERR_INVALID_CHARS, path,-1, NULL, 0, NULL, NULL); ^ git2r_cred.c: At top level: git2r_cred.c:209:12: warning: 'git2r_file_exists' defined but not used [-Wunused-function] static int git2r_file_exists(const char *path) ^ make: *** [git2r_cred.o] Error 1 Warning: running command 'make -f "Makevars.win" -f "C:/PROGRA~1/R/R-34~1.3/etc/x64/Makeconf" -f "C:/PROGRA~1/R/R-34~1.3/share/make/winshlib.mk" -f "C:\Users\JennyVM\AppData\Local\Temp\RtmpGqLLNN\file37c663237b7" SHLIB="git2r.dll" WIN=64 TCLBIN=64 OBJECTS="git2r.o git2r_S3.o git2r_arg.o git2r_blame.o git2r_blob.o git2r_branch.o git2r_checkout.o git2r_clone.o git2r_commit.o git2r_config.o git2r_cred.o git2r_diff.o git2r_error.o git2r_graph.o git2r_index.o git2r_libgit2.o git2r_merge.o git2r_note.o git2r_object.o git2r_odb.o git2r_oid.o git2r_push.o git2r_reference.o git2r_reflog.o git2r_remote.o git2r_repository.o git2r_reset.o git2r_revparse.o git2r_revwalk.o git2r_signature.o git2r_stash.o git2r_status.o git2r_tag.o git2r_transfer.o git2r_tree.o"' had status 2 ERROR: compilation failed for package 'git2r'
- removing 'C:/Users/JennyVM/AppData/Local/Temp/RtmpGqLLNN/devtools_install_37c68c712e1/git2r' In R CMD INSTALL Error in run(bin, args = real_cmdargs, stdout_line_callback = real_callback(stdout), : System command error
Now we're making progress! But still isn't returning path to my keys.
Installed from e6e2de5d23d0b91769ce0a38fc8be9ffbe83cf02
> .Call(git2r:::git2r_ssh_keys)
path: C:/Users/JennyVM/Documents\
path: C:\Users\JennyVM\
path: C:\Users\JennyVM\
NULL
Now it works! Installed from 5e8baea36896cf3a20b1d0d29510d6941b245b3d
> .Call(git2r:::git2r_ssh_keys)
private: C:\Users\JennyVM\.ssh/id_ed25519 public: C:\Users\JennyVM\.ssh/id_ed25519.pub
private: C:\Users\JennyVM\.ssh/id_rsa public: C:\Users\JennyVM\.ssh/id_rsa.pub
private: C:\Users\JennyVM\.ssh/id_ed25519 public: C:\Users\JennyVM\.ssh/id_ed25519.pub
private: C:\Users\JennyVM\.ssh/id_rsa public: C:\Users\JennyVM\.ssh/id_rsa.pub
NULL
Does it make sense that the keys are each showing up twice?
Note you've got a compilation warning now:
c:/Rtools/mingw_64/bin/gcc -I"C:/PROGRA~1/R/R-34~1.3/include" -DNDEBUG -I../windows/libgit2-0.27.2/include -DR_NO_REMAP -DSTRICT_R_HEADERS -O2 -Wall -std=gnu99 -mtune=generic -UNDEBUG -Wall -pedantic -g -Og -fdiagnostics-color=always -c git2r_cred.c -o git2r_cred.o
git2r_cred.c: In function 'git2r_ssh_keys':
git2r_cred.c:273:57: warning: passing argument 2 of 'git2r_path_from_environment_variable' discards 'const' qualifier from pointer target type
if (git2r_path_from_environment_variable(&path, env[i]))
^
git2r_cred.c:226:5: note: expected 'wchar_t *' but argument is of type 'const wchar_t *'
int git2r_path_from_environment_variable(char** out, wchar_t *env)
^
Great :smile: I will fix the duplicates and compiler warning. The next step is to integrate this code in the libgit2 callback.
On my machine, git2r now finds the default keys so that it works to push/pull/clone. Note, it doesn't handle passphrase protected keys yet (https://github.com/ropensci/git2r/blob/credentials/src/git2r_cred.c#L385).
With that branch I still need to write
cred <- git2r::cred_ssh_key(file.path(Sys.getenv("HOME"), ".ssh", "id_rsa.pub"), file.path(Sys.getenv("HOME"), ".ssh", "id_rsa"))
git2r::push(r, "origin", "refs/heads/master", credentials = cred)
@maelle thanks for testing. I have fixed to also handle passphrase protected keys using getPass. Does it work now?
It doesn't 😿 I installed the credentials branch ropensci/git2r@5c3659c and still need to first create cred and pass it as an argument to git2r::push