jj-vcs
jj git clone fails with ecdsa-sk hardware SSH key
Description
jj git clone [email protected]:martinvonz/jj.git fails if the user is using an SSH key with a
hardware security key such as the types described in Github's SSH key documentation.
Steps to Reproduce the Problem
- Create a new SSH key with
ssh-keygen -t ecdsa-sk -C "[email protected]"and a hardware security key. - Run
jj git clone [email protected]:martinvonz/jj.git. - See an error and the repo is not cloned.
Expected Behavior
The repo should be cloned.
Actual Behavior
jj git clone [email protected]:martinvonz/jj.git
Error: authentication required but no callback set; class=Ssh (23); code=Auth (-16)
Hint: Jujutsu uses libssh2, which doesn't respect ~/.ssh/config. Does `ssh -F /dev/null` to the host work?
Specifications
- Platform: MacOs
- Version:
jj 0.13.0
Sorry, the title was incorrect. It said ed25519-sk but this issue is actually for ecdsa-sk. In either case I think this is potentially different since it's the hardware security key version of the key. If fixing #2931 somehow fixes this issue as well that would be great though.
While this might be a libssh2 issue or something else, I want to mention that gpg-backed ssh keys work great :) incl. with a gpg keycard impl, such as yubikey (me) and presumably others
edit: oh wait, no idea about ecdsa, ec25519 are the ones working fine 🤷♂️ but not the -sk ones, the gpg ones
$ jj --version
jj 0.13.0
My experience is slightly different. I use ed25519-sk (so U2F backed by hardware dongle), though from openssh's perspective, the auth flow is exactly the same as ecdsa-sk, which is why I report here instead of opening a new ticket:
$ jj git clone [email protected]:martinvonz/jj.git
Error: Failed to authenticate SSH session: Unable to extract public key from private key file: Wrong passphrase or invalid/unrecognized private key file format; class=Ssh (23)
Note that the hardware dongle gets activated, requires a press (as configured/expected), then errors out as shown. There is no related config in .ssh/config. The similar git clone ... works just fine™.
Same as #2931?
That looks like a general configuration problem: OP reported RSA doesn't work either.
I've done a little investigation as I can't push using an ed25519-sk key either.
probably irrelevant packaging debugging to check that there is -sk support at all
From what I can see libssh2 added support for -sk in v1.11.0, the latest version of the libssh2-sys crate vendors libssh2 v1.10.0 and by default uses the vendored version instead of system version, but I'm using a nixpkgs build which disables the vendored libssh2, but this silently fails because of a missing zlib system package and fallsback to using the vendored version (silent until I added -vv to get log output from the build script), but even after patching that and confirming I have a version of jj built against libssh2 v1.11.0 it still fails.
Running with more logs I see that it is trying the ssh-agent first, and I see my security key request activation, but then presumably that fails to authenticate for some reason, then it sees the ssh-key and tries to use it but fails because no passphrase is passed
2024-03-02T00:15:57.702907Z DEBUG run:run_internal:run_command: git2::cred: executing credential helper "sh" "-c" "git credential-cache get"
2024-03-02T00:15:57.708990Z TRACE run:run_internal:run_command: git2::cred: credential helper stderr ---
2024-03-02T00:15:57.709008Z TRACE run:run_internal:run_command: git2::cred: ignoring output line:
2024-03-02T00:15:57.709048Z INFO run:run_internal:run_command: jj_lib::git: trying ssh_key_from_agent username="git"
2024-03-02T00:16:02.431408Z DEBUG run:run_internal:run_command: git2::cred: executing credential helper "sh" "-c" "git credential-cache get"
2024-03-02T00:16:02.436531Z TRACE run:run_internal:run_command: git2::cred: credential helper stderr ---
2024-03-02T00:16:02.436539Z TRACE run:run_internal:run_command: git2::cred: ignoring output line:
2024-03-02T00:16:02.436588Z INFO run:run_internal:run_command:get_ssh_keys{_username="git"}: jj_cli::git_util: found ssh key path="/home/nemo157/.ssh/id_ed25519_sk"
2024-03-02T00:16:02.436601Z INFO run:run_internal:run_command: jj_lib::git: trying ssh_key username="git" path="/home/nemo157/.ssh/id_ed25519_sk"
Error: Failed to authenticate SSH session: Unable to extract public key from private key file: Wrong passphrase or invalid/unrecognized private key file format; class=Ssh (23)
I built Jujutsu against a version of libssh2 with debug logging enabled (nix overlay)
pkgs-prev.jujutsu.overrideAttrs {
buildInputs = with pkgs-final; [
openssl
zstd
libgit2
zlib
(libssh2.overrideAttrs {
configureFlags = ["--enable-debug"];
patches = libssh2.patches ++ [
(writeText "log-everything.patch" ''
diff --git a/src/misc.c b/src/misc.c
index 7de6511b54...105a821b2f 100644
--- a/src/misc.c
+++ b/src/misc.c
@@ -526,11 +526,6 @@
const char *contexttext = contexts[0];
unsigned int contextindex;
- if(!(session->showmask & context)) {
- /* no such output asked for */
- return;
- }
-
/* Find the first matching context string for this message */
for(contextindex = 0; contextindex < ARRAY_SIZE(contexts);
contextindex++) {
'')
];
})
];
}
From those logs it appears to successfully send the sign-request to the ssh-agent, then forward the result to the server, which responds with an authentication failure. I also dumped the sign-request from the agent-side and compared to the sign-request sent by git, afaict they are using identical parameters, so it seems like that part is fine. I guess the next step would be to setup a test sshd-instance and try and see what's different about the auth request sent to the server that makes it get rejected.
EDIT: Manually reduced logs
-- without key in ssh-agent
[libssh2] 1.040309 Transport: Requesting userauth service
[libssh2] 1.162950 Socket: Recved 52/35000 bytes to 0x565154807028+0
[libssh2] 1.162976 Transport: Packet type 6 received, length=17
[libssh2] 1.164493 Socket: Sent 68/68 bytes at 0x56515480f920
[libssh2] 1.271050 Socket: Recved 52/35000 bytes to 0x565154807028+0
[libssh2] 1.271061 Transport: Packet type 51 received, length=15
[libssh2] 1.271070 Userauth: Permitted auth methods: publickey
[libssh2] 1.279698 Failure Event: 0 - agent list id failed
-- note, 0 == success, this isn't actually a failure
[libssh2] 1.279792 Socket: Sent 68/68 bytes at 0x56515480f920
[libssh2] 1.382265 Socket: Recved 52/35000 bytes to 0x565154807028+0
[libssh2] 1.382277 Transport: Packet type 51 received, length=15
[libssh2] 1.382287 Userauth: Permitted auth methods: publickey
[libssh2] 1.389153 Userauth: Computing public key from private key file: /home/nemo157/.ssh/id_ed25519_sk
[libssh2] 1.389897 Failure Event: -16 - Not an OpenSSH key file
[libssh2] 1.389901 Failure Event: -16 - Unable to extract public key from private key file: Wrong passphrase or invalid/unrecognized private key file format
Error: Failed to authenticate SSH session: Unable to extract public key from private key file: Wrong passphrase or invalid/unrecognized private key file format; class=Ssh (23)
-- with key in ssh-agent, but without hardware-key plugged in, so agent rejects signing
[libssh2] 0.843462 Transport: Requesting userauth service
[libssh2] 0.843475 Socket: Sent 52/52 bytes at 0x56217984cca0
[libssh2] 0.970409 Socket: Recved 52/35000 bytes to 0x5621798443a8+0
[libssh2] 0.970415 Transport: Packet type 6 received, length=17
[libssh2] 0.970453 Socket: Sent 68/68 bytes at 0x56217984cca0
[libssh2] 1.074018 Socket: Recved 52/35000 bytes to 0x5621798443a8+0
[libssh2] 1.074025 Transport: Packet type 51 received, length=15
[libssh2] 1.074032 Userauth: Permitted auth methods: publickey
[libssh2] 1.079691 Failure Event: 0 - agent list id failed
[libssh2] 1.079749 Key Ex: Signing using [email protected]
[libssh2] 1.079754 Userauth: Attempting publickey authentication
[libssh2] 1.079828 Socket: Sent 180/180 bytes at 0x56217984cca0
[libssh2] 1.188784 Socket: Recved 148/35000 bytes to 0x5621798443a8+0
[libssh2] 1.188791 Transport: Packet type 60 received, length=109
[libssh2] 1.196161 Failure Event: -42 - agent sign failure
[libssh2] 1.196165 Failure Event: -19 - Callback returned error
[libssh2] 1.196210 Socket: Sent 68/68 bytes at 0x56217984cca0
[libssh2] 1.299991 Socket: Recved 52/35000 bytes to 0x5621798443a8+0
[libssh2] 1.299998 Transport: Packet type 51 received, length=15
[libssh2] 1.300003 Userauth: Permitted auth methods: publickey
[libssh2] 1.305106 Userauth: Computing public key from private key file: /home/nemo157/.ssh/id_ed25519_sk
[libssh2] 1.305689 Failure Event: -16 - Not an OpenSSH key file
[libssh2] 1.305693 Failure Event: -16 - Unable to extract public key from private key file: Wrong passphrase or invalid/unrecognized private key file format
Error: Failed to authenticate SSH session: Unable to extract public key from private key file: Wrong passphrase or invalid/unrecognized private key file format; class=Ssh (23)
-- with key in agent and hardware-key triggered
[libssh2] 1.042866 Transport: Requesting userauth service
[libssh2] 1.042883 Socket: Sent 52/52 bytes at 0x5610bba7c380
[libssh2] 1.042885 Transport: Looking for packet of type: 6
[libssh2] 1.174748 Socket: Recved 52/35000 bytes to 0x5610bba73a88+0
[libssh2] 1.174769 Transport: Packet type 6 received, length=17
[libssh2] 1.174829 Socket: Sent 68/68 bytes at 0x5610bba7c380
[libssh2] 1.285124 Socket: Recved 52/35000 bytes to 0x5610bba73a88+0
[libssh2] 1.285141 Transport: Packet type 51 received, length=15
[libssh2] 1.285148 Userauth: Permitted auth methods: publickey
[libssh2] 1.291192 Failure Event: 0 - agent list id failed
[libssh2] 1.291203 Key Ex: Signing using [email protected]
[libssh2] 1.291208 Userauth: Attempting publickey authentication
[libssh2] 1.291312 Socket: Sent 180/180 bytes at 0x5610bba7c380
[libssh2] 1.405038 Socket: Recved 148/35000 bytes to 0x5610bba73a88+0
[libssh2] 1.405048 Transport: Packet type 60 received, length=109
-- hardware-key triggered
[libssh2] 3.557749 Failure Event: 0 - agent sign failure
[libssh2] 3.557763 Userauth: Attempting publickey authentication -- phase 2
[libssh2] 3.557880 Socket: Sent 276/276 bytes at 0x5610bba7c380
[libssh2] 3.667717 Socket: Recved 52/35000 bytes to 0x5610bba73a88+0
[libssh2] 3.667726 Transport: Packet type 51 received, length=15
[libssh2] 3.667737 Failure Event: -19 - Invalid signature for supplied public key, or bad username/public key combination
[libssh2] 3.667787 Socket: Sent 68/68 bytes at 0x5610bba7c380
[libssh2] 3.780792 Socket: Recved 52/35000 bytes to 0x5610bba73a88+0
[libssh2] 3.780799 Transport: Packet type 51 received, length=15
[libssh2] 3.780808 Userauth: Permitted auth methods: publickey
[libssh2] 3.786092 Userauth: Computing public key from private key file: /home/nemo157/.ssh/id_ed25519_sk
[libssh2] 3.786585 Failure Event: -16 - Not an OpenSSH key file
[libssh2] 3.786588 Failure Event: -16 - Unable to extract public key from private key file: Wrong passphrase or invalid/unrecognized private key file format
Error: Failed to authenticate SSH session: Unable to extract public key from private key file: Wrong passphrase or invalid/unrecognized private key file format; class=Ssh (23)
Are your keys passphrase-protected, and are you able to use/using the ssh-agent to store passphrases?
Yes, they're passphrase protected, which appears to be why the subsequent attempt to use the key directly fails; they're stored unlocked in ssh-agent.
Yeah, I think the libgit2/libssh2 integration isn't the best; I don't have the most experience with the technology. If you'd like, you can also try https://github.com/martinvonz/jj/pull/3191 which uses OpenSSH directly for creating the connection, which should probably work better with the ssh-agent. (It doesn't seem to work when run via Nix for me, so you might need to cargo build manually).
Yep, that branch appears to work fine. It can both use the key from the agent, and use it from disk with a passphrase prompt and auto-load it into the agent.