Cannot read remote files through ssh

[tshu-w]

As discussed in https://github.com/wakatime/wakatime-cli/issues/426, waketime can not read remote files through ssh. I tried to make a minimal reproduction sample in this issue.

Steps

1. ssh 124.16.139.208 -p 10722 login without password currectly.
2. wakatime-cli --entity ssh://124.16.139.208:10722/home/wangtianshu/test_project/test.py got error

Configs

~/.ssh/config: empty ~/.ssh/knwon_hosts:

[124.16.139.208]:10722 ssh-ed25519 AAAAC3***
[124.16.139.208]:10722 ssh-rsa AAAAB3***
[124.16.139.208]:10722 ecdsa-sha2-nistp256 AAAAE2***

Logs:

{"caller":"github.com/wakatime/wakatime-cli/pkg/remote/remote.go:88","func":"1","level":"error","message":"failed to download file to temporary folder: failed to connect to sftp host: failed to connecto to '124.16.139.208:10722': ssh: handshake failed: ssh: host key mismatch","now":"2022-02-23T10:35:58+08:00","version":"v1.37.0"}
{"caller":"github.com/wakatime/wakatime-cli/pkg/language/chroma.go:128","func":"selectByCustomizedPriority","level":"warning","message":"failed to load folder extensions: failed to read directory: open ssh://124.16.139.208:10722/home/wangtianshu/test_project/: no such file or directory","now":"2022-02-23T10:35:58+08:00","version":"v1.37.0"}

Environment:

• OS: mac
• Platform: 386

[gandarez]

It happens probabaly because you have multiple entries in ~/.ssh/knwon_hosts and wakatime-cli only take the first one. I'll implement a way where we take all of them and try each one before aborting.

[tshu-w]

@gandarez I tried to delete known_hosts and ssh, these three entries are both added.

[gandarez]

If you're using vscode you could try our latest alpha version by enabling alpha = true in ~/.wakatime.cfg.

[tshu-w]

Hi, this time I got unable to authenticate even if I can ssh without password (the authenticate error disappear if I pass password to wakatime-cli and my id_rsa is place at ~/.ssh/id_rsa).

{"caller":"/Users/runner/work/wakatime-cli/wakatime-cli/pkg/remote/remote.go:213","func":"Connect","level":"warning","message":"failed to connect to '124.16.139.208:10722': failed to dial to '124.16.139.208:10722': ssh: handshake failed: ssh: host key mismatch","now":"2022-02-24T09:31:18+08:00","version":"v1.39.0-alpha.1"}
{"caller":"/Users/runner/work/wakatime-cli/wakatime-cli/pkg/remote/remote.go:213","func":"Connect","level":"warning","message":"failed to connect to '124.16.139.208:10722': failed to dial to '124.16.139.208:10722': ssh: handshake failed: ssh: host key mismatch","now":"2022-02-24T09:31:18+08:00","version":"v1.39.0-alpha.1"}
{"caller":"/Users/runner/work/wakatime-cli/wakatime-cli/pkg/remote/remote.go:213","func":"Connect","level":"warning","message":"failed to connect to '124.16.139.208:10722': failed to dial to '124.16.139.208:10722': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain","now":"2022-02-24T09:31:19+08:00","version":"v1.39.0-alpha.1"}

[gandarez]

@tshu-w could you please enable debug = true in your ~/.wakatime.cfg, clean-up your ~/.wakatime.log file and do it again? Please send the log file to [email protected].

[alanhamlett]

@tshu-w so normally your ssh key isn't at ~/.ssh/id_rsa and you specify the key to use in ~/.ssh/config? Then the missing part is wakatime-cli needing to read ~/.ssh/config to find the correct private key file to use.

[alanhamlett]

the authenticate error disappear if I pass password to wakatime-cli and my id_rsa is place at ~/.ssh/id_rsa

Only password or key should be needed but not both. Does it work without password when your ssh key is at ~/.ssh/id_rsa?

[tshu-w]

@tshu-w could you please enable debug = true in your ~/.wakatime.cfg, clean-up your ~/.wakatime.log file and do it again? Please send the log file to [email protected].

Done. Please check you email.

Does it work without password when your ssh key is at ~/.ssh/id_rsa?

No, I cleared out my ~/.ssh/config and make sure there is a ~/.ssh/id_rsa which I can use to ssh, but I got above error.

[tshu-w]

I am not familiar with go, but I would like to ask about the code implementation. I understand if this kind of functionality should make use of existing tools as much as possible (by calling system related ssh tools for file caching etc.). But wakatime-cli seems to require parsing ~/.ssh/knwon_hosts by itself? And there is also a lack of support for ~/.ssh/config.

[alanhamlett]

We don't want to re-use existing ssh tools because that's system dependent. We're fine with using go libraries to parse these files, but that's not much different than parsing them ourselves.

[tshu-w]

In fact, it is much easier to use system specific ssh tools to get the file than to parse ssh config yourself. After all, we only have a few major categories of systems (Windows, Linux/macOS), and trying to parse the ssh config file is far more complicated than one might think, because there are so many options, such as the command Match needs to execute shell commands. Also, your PR doesn't seem to support common ProxyCommand, and my scenario is probably more complicated because I'm using the GPG key for ssh authentications. With all due respect, I don't see any hope of supporting it the current way.

[alanhamlett]

When we find ProxyCommand we can try executing the sftp binary to download the file, but in most cases I want to stay in Go-land because a lot of bugs are caused by depending or expecting things to be a certain way on every Linux machine.

[duskmoon314]

Hello. I came across a similar problem that the vsc extension cannot connect to the remote server and download files:

{"caller":"pkg/remote/remote.go:90",
"file":"ssh://7b22686f73744e616d65223a225034227d/root/path/to/file",
"func":"remote.WithDetection",
"level":"error",
"lineno":64,
"message":"failed to download file to temporary folder: failed to connect to sftp host: failed to connect to '7b22686f73744e616d65223a225034227d:22': failed to dial to '7b22686f73744e616d65223a225034227d:22': ssh: handshake failed: read tcp 198.18.0.1:53151->198.18.0.87:22: read: connection reset by peer",
"now":"2023-06-30T17:13:39+08:00",
"os/arch":"darwin/arm64",
"plugin":"vscode/1.79.2 vscode-wakatime/24.0.14",
"time":1688116419.511492,
"version":"v1.73.1"}

The IP 198.18.0.1 and 198.18.0.87 is due to a proxy's DNS hijack, which may not be a problem if the domain or real ip is correct. Thus, I'm curious where the 7b22686f73744e616d65223a225034227d comes from.

Here is my ssh config:

Host <target host name>
    HostName 166.111.***.***
    Port 5101
    User root

Since I have other hosts with similar config and they work well with wakatime. I wonder where to start to find out the cause of this problem.

[alanhamlett]

Could 7b22686f73744e616d65223a225034227d be an ipv6 address?

I think @tshu-w is right, we should fallback to system tools (sftp/ssh) to download remote files when downloading them using the pure-Go ssh client fails.

[duskmoon314]

Could 7b22686f73744e616d65223a225034227d be an ipv6 address?

I think the probability is low since the ssh config is using ipv4 and my proxy's dns hijacking translates ipv6 to v4 if not filtered.

I will check whether this could be ipv6 later. (Maybe Monday)

[duskmoon314]

The length of 7b22686f73744e616d65223a225034227d is two characters longer than the maximum of u128, so I think it could not be an ipv6 address.

[uniform641]

I met a similar problem due to ProxyCommand. Here is the debug log.

{"caller":"pkg/remote/remote.go:428","file":"ssh://108-33/home/uniform64/read-write-conflict/plot.py","func":"remote.Client","level":"debug","lineno":89,"message":"StrictHostKeyChecking for 108-33 set to ask","now":"2023-08-30T11:09:55+08:00","os/arch":"linux/amd64","plugin":"vscode/1.81.1 vscode-wakatime/24.2.2","time":1693364995.1014576,"version":"v1.76.0"}
{"caller":"pkg/remote/remote.go:286","file":"ssh://108-33/home/uniform64/read-write-conflict/plot.py","func":"remote.Client","level":"debug","lineno":89,"message":"failed to open known_hosts file: open ~/.ssh/known_hosts ~/.ssh/known_hosts2: no such file or directory","now":"2023-08-30T11:09:55+08:00","os/arch":"linux/amd64","plugin":"vscode/1.81.1 vscode-wakatime/24.2.2","time":1693364995.1014576,"version":"v1.76.0"}
{"caller":"pkg/remote/remote.go:286","file":"ssh://108-33/home/uniform64/read-write-conflict/plot.py","func":"remote.Client","level":"debug","lineno":89,"message":"failed to open known_hosts file: open /home/uniform64/.ssh/known_hosts ~/.ssh/known_hosts2: no such file or directory","now":"2023-08-30T11:09:55+08:00","os/arch":"linux/amd64","plugin":"vscode/1.81.1 vscode-wakatime/24.2.2","time":1693364995.1014576,"version":"v1.76.0"}
{"caller":"pkg/remote/remote.go:450","file":"ssh://108-33/home/uniform64/read-write-conflict/plot.py","func":"remote.Client","level":"debug","lineno":89,"message":"no known host key found for 108-33, will connect anyway","now":"2023-08-30T11:09:55+08:00","os/arch":"linux/amd64","plugin":"vscode/1.81.1 vscode-wakatime/24.2.2","time":1693364995.1014576,"version":"v1.76.0"}
{"caller":"pkg/remote/remote.go:90","file":"ssh://108-33/home/uniform64/read-write-conflict/plot.py","func":"heartbeat.initHandleOptions","level":"error","lineno":89,"message":"failed to download file to temporary folder: failed to connect to sftp host: failed to connect to '172.16.0.33:22': failed to dial to '172.16.0.33:22': dial tcp 172.16.0.33:22: i/o timeout","now":"2023-08-30T11:10:15+08:00","os/arch":"linux/amd64","plugin":"vscode/1.81.1 vscode-wakatime/24.2.2","time":1693364995.1014576,"version":"v1.76.0"}

And here is the corresponding configuration in ~/.ssh/config

Host 108-33
    HostName 172.16.0.33
    Port 22
    User uniform64
    ProxyCommand ssh -W %h:%p 108JumpServer

Wakatime-cli directly tried to connect to the ip address without proxy through 108JumpServer. I think that's the reason why it failed to download file to temporary folder.

[gandarez]

What do you think if we apply a regular expression ssh -W %h:%p (.*)? or ssh (.*)? -W %h:%p to extract the proxy server and try to connet to it before actually dialing to the final ssh host?

[duskmoon314]

What about ProxyJump? I recently faced this problem with another host with ProxyJump on a Windows machine.

I'm not familiar with Go. Is it easy to add a fallback to use the system tool?

[gandarez]

No, there isn't any easy way to manage it.

[gandarez]

@tshu-w @duskmoon314 we've just released v1.81.0 which added a fallback to attempt downloading the remote file.

[tshu-w]

@gandarez ~~I got the following log, does this indicate that the fallback successfully downloaded the file?~~ After reading the PR, I think the fallback work.

{"caller":"cmd/offline/offline.go:63","func":"offline.loadParams","level":"warning","message":"failed to load API parameters: api key not found or empty","now":"2023-09-15T09:38:25+08:00","os/arch":"darwin/arm64","version":"v1.81.0"}
{"caller":"pkg/remote/remote.go:92","file":"ssh://det/home/tianshu2020/DBCopilot/scripts/prepare_data.py","func":"offline.initHandleOptions","level":"error","message":"failed to download file to temporary folder: failed to connect to sftp host: failed to connect to 'det:22': failed to dial to 'det:22': ssh: handshake failed: EOF","now":"2023-09-15T09:38:26+08:00","os/arch":"darwin/arm64","time":1694741905.954981,"version":"v1.81.0"}
{"caller":"pkg/remote/remote.go:240","file":"ssh://det/home/tianshu2020/DBCopilot/scripts/prepare_data.py","func":"remote.Client","level":"info","message":"downloading remote file using fallback option","now":"2023-09-15T09:38:26+08:00","os/arch":"darwin/arm64","time":1694741905.954981,"version":"v1.81.0"}

[duskmoon314]

@gandarez I still see error messages when accessing a host through ProxyJump, both macOS and Windows 11.

{"caller":"pkg/remote/remote.go:92","file":"ssh://7b22686f73744e616d65223a225034227d/root/path/to/file","func":"heartbeat.initHandleOptions","level":"error","lineno":54,"message":"failed to download file to temporary folder: failed to connect to sftp host: failed to connect to '7b22686f73744e616d65223a225034227d:22': failed to dial to '7b22686f73744e616d65223a225034227d:22': dial tcp: lookup 7b22686f73744e616d65223a225034227d: no such host","now":"2023-09-15T11:18:01+08:00","os/arch":"darwin/arm64","plugin":"vscode/1.82.2 vscode-wakatime/24.2.2","time":1694747881.0097642,"version":"v1.81.0"}
{"caller":"pkg/remote/remote.go:240","file":"ssh://7b22686f73744e616d65223a225034227d/root/path/to/file","func":"remote.Client","level":"info","lineno":54,"message":"downloading remote file using fallback option","now":"2023-09-15T11:18:01+08:00","os/arch":"darwin/arm64","plugin":"vscode/1.82.2 vscode-wakatime/24.2.2","time":1694747881.0097642,"version":"v1.81.0"}
{"caller":"pkg/remote/remote.go:96","file":"ssh://7b22686f73744e616d65223a225034227d/root/path/to/file","func":"heartbeat.initHandleOptions","level":"error","lineno":54,"message":"failed to download remote file using fallback option: exit status 255","now":"2023-09-15T11:18:01+08:00","os/arch":"darwin/arm64","plugin":"vscode/1.82.2 vscode-wakatime/24.2.2","time":1694747881.0097642,"version":"v1.81.0"}

I haven't seen this problem on other hosts without ProxyJump for now.

[tshu-w]

@duskmoon314 FYI, the host I accessed through ProxyCommand is working fine. I haven't tried ProxyJump yet.

[alanhamlett]

After reading the PR, I think the fallback work.

@tshu-w yes but it looks like you're config file is missing an api key so your stats won't be uploaded.

I still see error messages when accessing a host through ProxyJump, both macOS and Windows 11.

@duskmoon314 yes, scp wasn't able to download the file failed to download remote file using fallback option: exit status 255

[gandarez]

@duskmoon314 do you always need password to connect to this host using ssh? If so, you need to add a ssh key to authorized_keys on remote host.

[duskmoon314]

I'm using passwords on some hosts and keys on others. In the above cases, I use an SSH key to connect.

[gandarez]

It should work with latest release where we added a fallback to scp if Golang's native library fails.