ssh2 Errors in hostKeyprove eventhough strictVendor is False
Hi All,
We are trying to connect with a server which fails after sending the below request (Authentication succeeds but fails in hostkeysprove)
Outbound: Sending GLOBAL_REQUEST ([email protected])
Tried disabling "strictVendor" check since the hostKeys were changed recently for the server but that doesnt seem to help. Looks like the verification occurs even though "strictVendor" is set to false. Referring to the below code in lib/client.js - function hostKeysProve
if (!client.config.strictVendor
|| (client.config.strictVendor && RE_OPENSSH.test(client._remoteVer))) {
Why would this be required even though strictVendor is false?
Also, tried disabling the below call
client._protocol.openssh_hostKeysProve(keys);
post disabling this we are able to connect fine with the server and also able to list directories inside it.
Not sure as to what is the issue here.
Adding the debug logs while failing to connect
INFO Try ssh2-sftp-client
INFO CLIENT[sftp]: connect: Debugging turned on
INFO CLIENT[sftp]: ssh2-sftp-client Version: 11.0.0 {
node: 18.20.4,
acorn: 8.11.3,
ada: 2.7.8,
ares: 1.28.1,
base64: 0.5.2,
brotli: 1.0.9,
cjs_module_lexer: 1.2.2,
cldr: 44.1,
icu: 74.2,
llhttp: 6.1.1,
modules: 108,
napi: 9,
nghttp2: 1.61.0,
openssl: 3.1.5,
simdutf: 5.2.4,
tz: 2024a,
undici: 5.28.4,
unicode: 15.1,
uv: 1.44.2,
uvwasi: 0.0.19,
v8: 10.2.154.26-node.37,
zlib: 1.3.0.1-motley
}
INFO CLIENT[sftp]: connect: Connect attempt 1
INFO Custom crypto binding not available
INFO Local ident: 'SSH-2.0-ssh2js1.16.0'
INFO Client: Trying <Server> on port 22 ...
INFO Socket connected
INFO Remote ident: 'SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.11'
INFO Outbound: Sending KEXINIT
INFO Inbound: Handshake in progress
INFO Handshake: (local) KEX method: [email protected],curve25519-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group15-sha512,diffie-hellman-group16-sha512,diffie-hellman-group17-sha512,diffie-hellman-group18-sha512,ext-info-c,[email protected]
INFO Handshake: (remote) KEX method: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,[email protected]
INFO Handshake: strict KEX mode enabled
INFO Handshake: KEX algorithm: [email protected]
INFO Handshake: (local) Host key format: ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-512,rsa-sha2-256,ssh-rsa
INFO Handshake: (remote) Host key format: ssh-ed25519,ecdsa-sha2-nistp256,ssh-ed25519
INFO Handshake: Host key format: ssh-ed25519
INFO Handshake: (local) C->S cipher: [email protected],[email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected]
INFO Handshake: (remote) C->S cipher: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
INFO Handshake: C->S Cipher: [email protected]
INFO Handshake: (local) S->C cipher: [email protected],[email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected]
INFO Handshake: (remote) S->C cipher: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
INFO Handshake: S->C cipher: [email protected]
INFO Handshake: (local) C->S MAC: [email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
INFO Handshake: (remote) C->S MAC: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
INFO Handshake: C->S MAC: <implicit>
INFO Handshake: (local) S->C MAC: [email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
INFO Handshake: (remote) S->C MAC: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
INFO Handshake: S->C MAC: <implicit>
INFO Handshake: (local) C->S compression: none,[email protected],zlib
INFO Handshake: (remote) C->S compression: none,[email protected]
INFO Handshake: C->S compression: none
INFO Handshake: (local) S->C compression: none,[email protected],zlib
INFO Handshake: (remote) S->C compression: none,[email protected]
INFO Handshake: S->C compression: none
INFO Outbound: Sending KEXECDH_INIT
INFO Received DH Reply
INFO Host accepted by default (no verification)
INFO Host accepted (verified)
INFO Outbound: Sending NEWKEYS
INFO Inbound: NEWKEYS
INFO Verifying signature ...
INFO Verified signature
INFO Handshake completed
INFO Outbound: Sending SERVICE_REQUEST (ssh-userauth)
INFO Inbound: Received EXT_INFO
INFO Inbound: Received SERVICE_ACCEPT (ssh-userauth)
INFO Outbound: Sending USERAUTH_REQUEST (none)
INFO Inbound: Received USERAUTH_FAILURE (publickey,password)
INFO Client: none auth failed
INFO Outbound: Sending USERAUTH_REQUEST (password)
INFO Inbound: Received USERAUTH_SUCCESS
INFO Outbound: Sending CHANNEL_OPEN (r:0, session)
INFO Inbound: GLOBAL_REQUEST ([email protected])
INFO Outbound: Sending GLOBAL_REQUEST ([email protected])
INFO Inbound: CHANNEL_OPEN_CONFIRMATION (r:0, s:0)
INFO Outbound: Sending CHANNEL_REQUEST (r:0, subsystem: sftp)
INFO Socket ended
INFO CLIENT[sftp]: connect endListener - handling unexpected end event
ERROR Uncaught Exception {errorType:Error,errorMessage:connect: Unexpected end event,stack:[Error: connect: Unexpected end event, at Client.fn (/var/task/node_modules/ssh2-sftp-client/src/utils.js:90:22), at Client.emit (node:events:529:35), at Socket.<anonymous> (/var/task/node_modules/ssh2/lib/client.js:812:12), at Socket.emit (node:events:529:35), at endReadableNT (node:internal/streams/readable:1400:12), at process.processTicksAndRejections (node:internal/process/task_queues:82:21)]}
Unknown application error occurred
Runtime.Unknown
Below are the logs after disabling "openssh_hostKeysProve" call
INFO Try ssh2-sftp-client
INFO CLIENT[sftp]: connect: Debugging turned on
INFO CLIENT[sftp]: ssh2-sftp-client Version: 11.0.0 {
node: 18.20.4,
acorn: 8.11.3,
ada: 2.7.8,
ares: 1.28.1,
base64: 0.5.2,
brotli: 1.0.9,
cjs_module_lexer: 1.2.2,
cldr: 44.1,
icu: 74.2,
llhttp: 6.1.1,
modules: 108,
napi: 9,
nghttp2: 1.61.0,
openssl: 3.1.5,
simdutf: 5.2.4,
tz: 2024a,
undici: 5.28.4,
unicode: 15.1,
uv: 1.44.2,
uvwasi: 0.0.19,
v8: 10.2.154.26-node.37,
zlib: 1.3.0.1-motley
}
INFO CLIENT[sftp]: connect: Connect attempt 1
INFO Custom crypto binding not available
INFO Local ident: 'SSH-2.0-ssh2js1.16.0'
INFO Client: Trying <Server> on port 22 ...
INFO Socket connected
INFO Remote ident: 'SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.11'
INFO Outbound: Sending KEXINIT
INFO Inbound: Handshake in progress
INFO Handshake: (local) KEX method: [email protected],curve25519-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group15-sha512,diffie-hellman-group16-sha512,diffie-hellman-group17-sha512,diffie-hellman-group18-sha512,ext-info-c,[email protected]
INFO Handshake: (remote) KEX method: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,[email protected]
INFO Handshake: strict KEX mode enabled
INFO Handshake: KEX algorithm: [email protected]
INFO Handshake: (local) Host key format: ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-512,rsa-sha2-256,ssh-rsa
INFO Handshake: (remote) Host key format: ssh-ed25519,ecdsa-sha2-nistp256,ssh-ed25519
INFO Handshake: Host key format: ssh-ed25519
INFO Handshake: (local) C->S cipher: [email protected],[email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected]
INFO Handshake: (remote) C->S cipher: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
INFO Handshake: C->S Cipher: [email protected]
INFO Handshake: (local) S->C cipher: [email protected],[email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected]
INFO Handshake: (remote) S->C cipher: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
INFO Handshake: S->C cipher: [email protected]
INFO Handshake: (local) C->S MAC: [email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
INFO Handshake: (remote) C->S MAC: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
INFO Handshake: C->S MAC: <implicit>
INFO Handshake: (local) S->C MAC: [email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
INFO Handshake: (remote) S->C MAC: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
INFO Handshake: S->C MAC: <implicit>
INFO Handshake: (local) C->S compression: none,[email protected],zlib
INFO Handshake: (remote) C->S compression: none,[email protected]
INFO Handshake: C->S compression: none
INFO Handshake: (local) S->C compression: none,[email protected],zlib
INFO Handshake: (remote) S->C compression: none,[email protected]
INFO Handshake: S->C compression: none
INFO Outbound: Sending KEXECDH_INIT
INFO Received DH Reply
INFO Host accepted by default (no verification)
INFO Host accepted (verified)
INFO Outbound: Sending NEWKEYS
INFO Inbound: NEWKEYS
INFO Verifying signature ...
INFO Verified signature
INFO Handshake completed
INFO Outbound: Sending SERVICE_REQUEST (ssh-userauth)
INFO Inbound: Received EXT_INFO
INFO Inbound: Received SERVICE_ACCEPT (ssh-userauth)
INFO Outbound: Sending USERAUTH_REQUEST (none)
INFO Inbound: Received USERAUTH_FAILURE (publickey,password)
INFO Client: none auth failed
INFO Outbound: Sending USERAUTH_REQUEST (password)
INFO Inbound: Received USERAUTH_SUCCESS
INFO Outbound: Sending CHANNEL_OPEN (r:0, session)
INFO Inbound: GLOBAL_REQUEST ([email protected])
INFO Inbound: CHANNEL_OPEN_CONFIRMATION (r:0, s:0)
INFO Outbound: Sending CHANNEL_REQUEST (r:0, subsystem: sftp)
INFO Inbound: CHANNEL_WINDOW_ADJUST (r:0, 2097152)
INFO Inbound: CHANNEL_SUCCESS (r:0)
INFO Outbound: Sending CHANNEL_DATA (r:0, 9)
INFO Inbound: CHANNEL_DATA (r:0, 183)
INFO SFTP: Inbound: Received VERSION (v3, exts:[email protected],[email protected],[email protected],[email protected],[email protected],[email protected])
INFO Connected
INFO Outbound: Sending CHANNEL_DATA (r:0, 14)
INFO SFTP: Outbound: Buffered OPENDIR
INFO Inbound: CHANNEL_DATA (r:0, 17)
INFO SFTP: Inbound: Received HANDLE (id:0)
INFO Outbound: Sending CHANNEL_DATA (r:0, 17)
INFO SFTP: Outbound: Buffered READDIR
INFO Inbound: CHANNEL_DATA (r:0, 323)
INFO SFTP: Inbound: Received NAME (id:1, 3)
INFO Outbound: Sending CHANNEL_DATA (r:0, 17)
INFO SFTP: Outbound: Buffered READDIR
INFO Inbound: CHANNEL_DATA (r:0, 32)
INFO SFTP: Inbound: Received STATUS (id:2, 1, End of file)
INFO Outbound: Sending CHANNEL_DATA (r:0, 17)
INFO SFTP: Outbound: Buffered CLOSE
INFO Inbound: CHANNEL_DATA (r:0, 28)
INFO SFTP: Inbound: Received STATUS (id:3, 0, Success)
INFO Directory: [object Object] <able to list directory contents>
INFO CLIENT[sftp]: end: Ending SFTP connection
INFO Outbound: Sending DISCONNECT (11)
INFO Socket ended
INFO CLIENT[sftp]: end endListener - ignoring handled end event
INFO CLIENT[sftp]: Global end event: Ignoring handled end event
INFO Socket closed
INFO CLIENT[sftp]: end: Connection closed
INFO CLIENT[sftp]: end closeListener - ignoring handled close event
INFO CLIENT[sftp]: Global close event: Ignoring handled close event
INFO End ssh2-sftp-client
Below is the code that was tried
const SFTPClient = require("ssh2-sftp-client");
let client = new SFTPClient();
console.log("Try ssh2-sftp-client");
client.connect({
host: "host",
port: 22,
username: "user",
password: "abcd",
strictVendor: false,
putDir: "/",
debug: (message) => { console.log(message) },
}).then(() => {
console.log("Connected");
return client.list('/');
}).then((data) => {
console.log('Directory: '+ data);
return client.end();
}).catch(err => {
console.log(`Error: ${err.message}`);
});
console.log("End ssh2-sftp-client");
Note: using a wrapper "ssh2-sftp-client" for connecting but raising this here since this error seems to occur from ssh2 library.
Why would this be required even though strictVendor is false?
strictVendor is about client-initiated requests. hostkeys-prove-00 is a response to hostkeys-00 server-initiated requests.
Are you able to generate and post (unused) server host keys that replicate the issue? I've never seen this be a problem before.
This code runs in AWS lambda hence we dont store host keys.
Since its lambda, a hostKey change at the server side shouldn't have affected the connectivity. Its to be noted that prior to the hostkey change the same code was able to connect with the server without any issues but post change we are facing this issue (even though host keys shouldn't have been stored/used)
Also, when i tried to do normal sftp from a linux server i was able to connect with the host successfully. Let me know if you would need that debug logs, if that will help.
Any idea why would this "hostkeys-00 server-initiated requests" doesnt cause any issues in a linux server but cause "Unexpected end event" when it runs via nodejs? Is there any way/config to skip "hostkeys-00" server requests from client side?
This code runs in AWS lambda hence we dont store host keys
I was referring to the host keys of the server you're connecting to not from. Either way, it doesn't need to be lambda or anything else, if you're able to generate new (unused) host keys for a server that you're in control of that still causes this error, then I can look into it. Otherwise I have no way to duplicate the issue on my end as I have not encountered it before.
but cause "Unexpected end event" when it runs via nodejs
I'm pretty sure that is not coming from ssh2, so I cannot help you there. Try using ssh2 directly and that error should not occur.
I did try with ssh2 directly and it also errors out. Below are the logs
INFO Try ssh2
INFO Custom crypto binding not available
INFO Local ident: 'SSH-2.0-ssh2js1.16.0'
INFO Client: Trying <Server> on port 22 ...
INFO Socket connected
INFO Remote ident: 'SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.11'
INFO Outbound: Sending KEXINIT
INFO Inbound: Handshake in progress
INFO Handshake: (local) KEX method: [email protected],curve25519-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group15-sha512,diffie-hellman-group16-sha512,diffie-hellman-group17-sha512,diffie-hellman-group18-sha512,ext-info-c,[email protected]
INFO Handshake: (remote) KEX method: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,[email protected]
INFO Handshake: strict KEX mode enabled
INFO Handshake: KEX algorithm: [email protected]
INFO Handshake: (local) Host key format: ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-512,rsa-sha2-256,ssh-rsa
INFO Handshake: (remote) Host key format: ssh-ed25519,ecdsa-sha2-nistp256,ssh-ed25519
INFO Handshake: Host key format: ssh-ed25519
INFO Handshake: (local) C->S cipher: [email protected],[email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected]
INFO Handshake: (remote) C->S cipher: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
INFO Handshake: C->S Cipher: [email protected]
INFO Handshake: (local) S->C cipher: [email protected],[email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected]
INFO Handshake: (remote) S->C cipher: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
INFO Handshake: S->C cipher: [email protected]
INFO Handshake: (local) C->S MAC: [email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
INFO Handshake: (remote) C->S MAC: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
INFO Handshake: C->S MAC: <implicit>
INFO Handshake: (local) S->C MAC: [email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
INFO Handshake: (remote) S->C MAC: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
INFO Handshake: S->C MAC: <implicit>
INFO Handshake: (local) C->S compression: none,[email protected],zlib
INFO Handshake: (remote) C->S compression: none,[email protected]
INFO Handshake: C->S compression: none
INFO Handshake: (local) S->C compression: none,[email protected],zlib
INFO Handshake: (remote) S->C compression: none,[email protected]
INFO Handshake: S->C compression: none
INFO Outbound: Sending KEXECDH_INIT
INFO Received DH Reply
INFO Host accepted by default (no verification)
INFO Host accepted (verified)
INFO Outbound: Sending NEWKEYS
INFO Inbound: NEWKEYS
INFO Verifying signature ...
INFO Verified signature
INFO Handshake completed
INFO Outbound: Sending SERVICE_REQUEST (ssh-userauth)
INFO Inbound: Received EXT_INFO
INFO Inbound: Received SERVICE_ACCEPT (ssh-userauth)
INFO Outbound: Sending USERAUTH_REQUEST (none)
INFO Inbound: Received USERAUTH_FAILURE (publickey,password)
INFO Client: none auth failed
INFO Outbound: Sending USERAUTH_REQUEST (password)
INFO Inbound: Received USERAUTH_SUCCESS
INFO Client :: ready
INFO Outbound: Sending CHANNEL_OPEN (r:0, session)
INFO Inbound: GLOBAL_REQUEST ([email protected])
INFO Outbound: Sending GLOBAL_REQUEST ([email protected])
INFO Inbound: CHANNEL_OPEN_CONFIRMATION (r:0, s:0)
INFO Outbound: Sending CHANNEL_REQUEST (r:0, subsystem: sftp)
INFO Socket ended
INFO Socket closed
ERROR Uncaught Exception {errorType:Error,errorMessage:Unable to start subsystem: sftp,stack:[Error: Unable to start subsystem: sftp, at Array.<anonymous> (/var/task/node_modules/ssh2/lib/client.js:1868:12), at onCHANNEL_CLOSE (/var/task/node_modules/ssh2/lib/utils.js:76:21), at ChannelManager.cleanup (/var/task/node_modules/ssh2/lib/utils.js:200:7), at Socket.<anonymous> (/var/task/node_modules/ssh2/lib/client.js:829:21), at Socket.emit (node:events:517:28), at TCP.<anonymous> (node:net:350:12)]}
Unknown application error occurred
Runtime.Unknown
Below is the code i tried
const conn = new Client();
conn.on('ready', () => {
console.log('Client :: ready');
conn.sftp((err, sftp) => {
if (err) throw err;
console.log('connected');
});
}).connect({
host: "server",
port: 22,
username: "user",
password: "abcd",
debug: console.log
});
Here too, if i comment that openssh host keys prove i dont see any errors. Also, when tried from linux server we are able to sftp into server and do ls / upload files into it as well.
it also errors out
Right, but now it's an 'uncaught exception' because you have an explicit throw err in there.
Also, when tried from linux server we are able to sftp into server and do ls / upload files into it as well.
I'm not sure what you're saying here, but I'm going to assume you mean the OpenSSH client when you say "linux server." I understand that that works. However as I said, unless I have some way to replicate the issue on my side, there's nothing I can do to fix the apparent bug.
yeah i meant the below sftp command works on the same server.
sftp -vvv user@server
Meanwhile will check if i can get the host keys. And I believe by host keys you mean the host fingerprint that gets stored in known_hosts file.
And I believe by host keys you mean the host fingerprint that gets stored in known_hosts file.
What I need is a newly generated (e.g. via ssh-keygen), unused host key (the private key itself, not the fingerprint or public key) that causes the problem so that I can duplicate the problem on my side and use the key to create a unit test.
we are using password auth to connect with the server not key based auth. So how does privatekey help here? Also this sftp server is not something we own
we are using password auth to connect with the server not key based auth
host keys, nothing to do with user authentication
Also this sftp server is not something we own
That makes things more difficult...
Correct me if i am wrong. So you wanted to do a ssh-keygen from within the sftp server and share the privateKey part of it?
So you wanted to do a ssh-keygen from within the sftp server and share the privateKey part of it?
Doesn't matter where it's generated, as long as it reproduces the problem you're having.