Allow using a SSH config file

[rgarrigue]

Hello

We've internal servers behind a bastion. Meaning, we can't resolve *.internal.company.com, instead our config define a ProxyJump on bounce.company.com for the internal domain. And once connected on the bastion, can resolve internal and go on connecting.

So either you add the ProxyJump instruction, but that'ld be a bad solution since there are more instructions required like ForwardAgent. Or your allow to define the command used, hence ssh -o ProxyJump bounce blabla; ForwardAgent yes; etc. But that'ld be a pain in the ass to redefine the whole config in VSCode. Or you allow reading a SSH config file.

Best regards

[SchoofsKelvin]

Some settings (e.g. ForwardAgent) don't make complete sense to use over SFTP? For your case, SSH hopping (#7) seems like a solution, as I send the host (without resolving it) to the hop to connect to. As far as I know, it should be the hop resolving that. The same should happen for HTTP (#64) and SOCK4/SOCK5 (#4) proxies.

Also worth noting: Even with proxying, the authentication and everything is done within vscode. If you specified the keys in the configs, and/or use the agent system, it should work. No need for ForwardAgent.

I'm currently working on a way to have external config files (#81), and afterwards, I'm planning on parsing (auto-detected) .ssh/config files, although not all options will be supported.

[rgarrigue]

ForwardAgent makes sense, since anyone in my company is given access to servers via its SSH keys.

SSH Hoping is the Proxyjump I mentioned

Good to hear the feature is on it's way. Thanks !

[SchoofsKelvin]

I started working on this feature, and pushed it to the feature/openssh branch. It'll still take a while before it's finished, as there's quite a bit to do:

• Make sure it can properly parse the configs
  • Finish the Match keyword's conditions (final/canonical/host with hostname canonicalization)
  • Have the Include work in the same way that OpenSSH does it (including globs, which'll be fun)
  • Parse more keywords (e.g. ProxyCommand/ProxyJump)
• Update the Settings UI to support linking/importing/... OpenSSH configs and config files
  • As a small bonus, also allow linking/importing PuTTY sessions, besides just the PuTTY config field
• Make the extension properly calculate configs that are based on OpenSSH configs
  • Similar to how PuTTY configs are handled, so not really a problem. Maybe intersection, though?
• Make the extension properly use the calculate config (e.g. ProxyJump allows multiple jumps)

Quite a bit indeed. Although it shouldn't take too long to have a basic "demo" working.

This might also, sooner or later, allow the working of StrictHostKeyChecking (#89)

[jhit]

I would love to see this finished. I have a loooong ssh config file and it would make using your extension so much easier.

[inetknght]

Hi,

I've got a lengthy ~/.ssh/config file which I'd rather not have to duplicate in this extension's config. I'd like to be able to refer to the hosts by their name. A workaround I've found is to mount the remote filesystem using sshfs but that uses FUSE which isn't available (or permitted) everywhere.

Thanks, @inetknght

[SchoofsKelvin]

I've started working on this in the feature/ssh-config branch. Last pushed commit will automatically get build. Here is a list of the builds for that branch.

I've only very very briefly tested it on my Windows 10 machine. It lacks a lot of features, and I'm sure there might be bugs or unexpected/weird interactions happening. This is also why I'm "releasing" it separately from the official releases for now.

For the commit I just pushed, supported options can be seen here. It can handle Host with (negate) patterns, and should support Match mostly.

To make use of the feature, open your settings.json (wherever your SSH FS config is stored) and add "sshConfig": true to it, like this:

"sshfs.configs": [
	{
		"name": "hetzner-ssh",
		"label": "Hetzer (SSH)",
		"host": "hetzner",
		"sshConfig": true
	}
]

Similar to OpenSSH, it'll use the given host (or prompt you if it's missing) and username (or your local username if it's missing) to check against the Host and Match directives. If one Host/Match has e.g. Hostname example.com, then the configs coming afterwards will check for example.com instead of the original hostname. Match originalhost ... aside, of course.

[inetknght]

That's pretty cool. I see you don't yet support ProxyJump which is one of the things I'm specifically needing. Consider a config file laid out like this:

# only present specified keys instead of all keys unlocked in the agent
# this prevents the server from thinking I'm trying to brute-force my way in
# I have thousands of keys (one for each machine in the fleet) and ssh tries each of them sequentially until one succeeds...
# so this tells SSH to only use the key I specify
IdentitiesOnly yes

# automatic agent forwarding is a bad practice. caching unlocked keys by default is a bad practice.
# i will set these with eg `ssh -o ForwardAgent yes` if I need to
ForwardAgent no 
IdentityAgent none
AddKeysToAgent no

# workstation is a phy computer running elsewhere.
# workstation.inetknght.dev cannot be resolved outside of the LAN
Host workstation
  Hostname workstation.inetknght.dev
  Port 55555
  LocalForward 127.0.0.1:8080 127.0.0.1:8080 # connecting to my localhost on port 8080 will actually connect to the workstation on the same port
  User inetknght
  IdentityFile /home/inetknght/.ssh/inetknght-localhost-inetknght-remote.id_ed25519 # non-default key

# develop is a VM running on the workstation. it hosts source code and build tools.
# it is often nuked and rebuilt using vagrant on the workstation.
Host develop
  # 192.168.56.x is the default address space for Virtual Box host-only NICs btw
  Hostname 192.168.56.18
  Port 44444
  User developer
  IdentityFile /home/inetknght/.ssh/inetknght-localhost-developer-develop.id_ed25519
  ProxyJump workstation
  ForwardX11 yes
  ForwardX11Trusted yes

In that configuration, to connect to develop I must first proxy through workstation. When I invoke ssh develop, I will be asked for the passphrase to unlock the first key (to workstation) and then the passphrase to unlock the second key (to develop).

When I use ssh to connect to develop, I am able to run X11 programs on the remote computer and have them display here. It would be nice to be able to build-and-run those programs in the same X11 session as VS Code but not strictly necessary. I'm most eager to be able to access the remote filesystem from within VS Codium without using sshfs.

SSH also has the option to specify the path to the config file. For example I have an SSH config file and a vscode settings file both committed to a repository. I'd like to have the vscode config use the SSH config in the repo instead of in my home dir.

It looks like you're parsing the SSH config with code you wrote yourself. I'm not familiar with Javascript or Typescript but if I understand correctly, they're somewhat compatible. Is there a reason you wrote it yourself instead of using a library? The ssh-config library from cyjake appears to be fairly clean even though it's javascript instead of typescript.

[SchoofsKelvin]

Regarding IdentitiesOnly in your config: The underlying ssh2 library I use only allows me to specify a single key (privateKey, which is also used for the hostbased user authentication), so my extension would only end up using the first specified IdentityFile, and (for now) actually ignores extra IdentityFiles or even the default paths. I'll look into whether I can "merge" private keys together and pass that to the library, although that sounds unlikely.

I see you don't yet support ProxyJump which is one of the things I'm specifically needing.

Part of the problem is that I have yet to rework the connection system to allow ad-hoc or "config-less" connections. While the extension supports SSH hopping, it requires you to specify another config to use. This rework would also support connecting to user@domain instantaneously without requiring a (full) config, which would allow me to easily support ProxyJump. (emulate hop with a partial config with sshConfig set to true and host set to whatever ProxyJump is set to)

It would be nice to be able to build-and-run those programs in the same X11 session as VS Code but not strictly necessary.

It's also not a priority in my eyes, as this extension is mostly focused on the FS and terminal support. Since I'm parsing ssh_config files that specifies X11 forwarding, and it seems relatively easily in the underlying library I use, I'm thinking about supporting it. I'd still have to test how it would behave, especially when a user opens several terminals and runs X11 programs on them.

SSH also has the option to specify the path to the config file. For example I have an SSH config file and a vscode settings file both committed to a repository. I'd like to have the vscode config use the SSH config in the repo instead of in my home dir.

I did add a sshfs.paths.ssh config option (ea33f106) that defaults to ["$HOME/.ssh/config", "/etc/ssh/ssh_config"] but could be overridden in your workspace file to point wherever.

It looks like you're parsing the SSH config with code you wrote yourself. I'm not familiar with Javascript or Typescript but if I understand correctly, they're somewhat compatible. Is there a reason you wrote it yourself instead of using a library? The ssh-config library from cyjake appears to be fairly clean even though it's javascript instead of typescript.

I actually did take a look at it, but decided against it. The main issues I had with it are that it doesn't support Match and isn't case insensitive. It also lacks TS typings, although not that big of an issue, although I noticed their values can be strings or arrays, depending on the directive, hardcoded in their code. I decided to write a simple parser myself, along with the logic for merging/computing configs. This allows me to add support for Match (and eventually Include), along with some other benefits. Bad sides of my implementation aside, I felt it was worth it. Aside from Match and parts of the warn/error reporting, it was relatively easy and quick to write anyway

[alexweej]

Randomly just hit this. Silly corporate ssh setup at work getting in the way. Thanks for your work on this issue.

[SchoofsKelvin]

Quick update: Just added instant-connections (844b0e1), which will help a lot with making ProxyJump work. I've also looked a bit into stuff, and supporting ProxyCommand should be relatively easy.

[SchoofsKelvin]

Changelog since the last major push:

1. Merged master branch (v1.20.0) back into the feature branch, to make use of new features and bug fixes This branch originated from v1.19.1, so missed quite some stuff you might want to have if daily-driving this branch
2. Connection strings can now start with ssh://, scp:// and sftp://, in preparation for ProxyJump support (18b80c5)
3. The extension now supports defining multiple hops in one config, e.g. "hops": ["hop1", "hop2"] (d5e6a2c)
4. The extension now supports a new proxy type command, which acts like ProxyCommand (f387681)
5. When using SSH configs, it will now also link ProxyJump and ProxyCommand to 3. and 4. (01d8b20) Note: The extension chooses hops over proxy (and thus ProxyJump over ProxyCommand). If your configuration somehow has hops set (by ProxyJump or somewhere else), it will ignore the ProxyCommand. Unlike how OpenSSH does it, the extension does not pick ProxyJump/ProxyCommand depending on which appears first in your configuration file(s), the extension will pick (the first) ProxyJump over any ProxyCommand. (shouldn't matter much, unless you have a Match all\n ProxyCommand ... "default" at the end of your file or so)

The current version can be downloaded here, built from this run. Unpack and drag the .vsix file into the Extensions view in VS Code. Sometimes I push commits without posting an update here, in which case you can still download the most up-to-date versions from this list, or clone the repository and build it yourself.

[SchoofsKelvin]

In the meantime, port forwarding is currently available as a beta build on the feature/forwarding branch. It basically offers the full functionality of LocalForward, RemoteForward and DynamicForward (even supporting their syntaxes in the config file), but is (currently) on a branch separate from feature/ssh-config, so don't expect it to work there yet.

[peey]

This feature is found in https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh but I guess it's not open source.

Still, it'd be a good idea to take inspiration about the UI offered to the developer from there.

Another way to solve this might be to just not make assumptions about what is and isn't required. If I have an ssh configuration called server-1 and I mention server-1 in the host, I shouldn't be necessarily prompted for username and password because it's possible that it's already specified in the ssh config. It should only be prompted if after using the sshfs command, the command itself prompts us for it.

[SchoofsKelvin]

This feature is found in https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh but I guess it's not open source.

From what I've seen/heard, it basically also just parses the ~/.ssh/config (or $HOMEDRIVE$HOMEPATH\.ssh\config on Windows, which is basically the same), similar to what I'm doing in the feature/ssh-config branch.

Another way to solve this might be to just not make assumptions about what is and isn't required. If I have an ssh configuration called server-1 and I mention server-1 in the host, I shouldn't be necessarily prompted for username and password because it's possible that it's already specified in the ssh config. It should only be prompted if after using the sshfs command, the command itself prompts us for it.

The extension is the one that is connecting (through the underlying ssh2 library), it doesn't make use of the ssh, plink or sshfs commands, everything happens in NodeJS (with the exception of a small .exe file used to read PuTTY configs from the Windows registry. Regardless, with the current demo in the feature/ssh-config branch, you already don't need to enter a username/password if sshConfig is set to true. It'll only use the username to filter configs (e.g. Match user my-user host myhost.example) but if you don't specify a username, it'll just match more configs and use the username from the first User my-user directive.

The config resolution currently looks like this:

• User attempts to open ssh://server-1/some/file
• Extension tries to find/generate a config for server-1
  • If a config with that exact name exists, return it
  • Parse the string to a config, e.g. user@server-1 to { username: 'user', host: 'server-1' }
  • If we find a config where the name equals the parsed host (e.g. server-1, return a merged version
    • Copy registered config and overwrite the fields that are present in the parsed config
    • Exception: for name we use user@host:port, where user and :port can be optional
      • E.g. ssh://server-1/ would actually result in the name @server-1, the @ meaning "this is an instant config"
    • Exception: for host we prefer the original version if present (since the parsed "host" might be a config name)
  • Return the parsed config
• At this point, we have a "registered" config (which might be an "instant" config)
  • This is also what we check in case want to check if a connection's config is outdated
• We now start to create an actual config (and connection) for this "initial config"
  • We normalize a bunch of things, e.g. a username of $USERNAME becomes $USER
  • We replace environment variables for certain fields, e.g. username, host, port, agent, private key path, ...
  • If the config has putty set (which is enabled by default for instant configs) we do the following:
    • We load all PuTTY sessions from registry and temporarily cache them for fast querying
    • If this is an instant connection, we try to find a PuTTY session with the same host
      • We do this since host might be the PuTTY config name, e.g. ssh://user@some-putty-session-name/
    • In case that failed or was skipped:
      • Prompt for the host field in case it's missing in the config
      • If putty is true, we try to find a session with the same name OR the same host/username (if present)
      • If putty is a string, we look for a session by name, but ignore it if the host/username mismatch (if present)
    • If we found a PuTTY session, we will:
      • Validate the PuTTY session is a SSH session (since PuTTY also supports Telnet, Serial, ...)
      • Copy the username/host/port/agent/privateKeyPath fields if present
        • For host, if it contains a @, we only keep everything after the first @
        • If the config has no username and the PuTTY session's host contains @, copy the user from before the @
      • Try to copy proxy settings from the PuTTY session (fail if not Socks 4/5 and not HTTP)
    • If we require a PuTTY session (e.g. putty is set and this isn't an instant connection) but don't have one, error
  • If the username is $USER at this point, replace it with what NodeJS's os module thinks is the current user's username
  • If sshConfig is set in the config, we try to parse OpenSSH config files
    • We prompt for the host field if it isn't present yet (any ssh command, i.e. OpenSSH, requires at least the host)
    • We read the sshfs.paths.ssh setting to find all config files (defaults to ["~/.ssh/config", "/etc/ssh/ssh_config"])
    • We read all these config files after replacing leading ~ and environment variables with their correct values
      • We read every registered config file in the same order as sshfs.paths.ssh dictates
      • We go through all directives, parse Host ..., Match ..., IdentityFile ..., User ..., ... basically all that we can
      • We also keep track of parsing warnings/errors, e.g. we don't support Match canonical
      • We save every config (in order), including duplicate Host hostname entries
    • We use these configs to build a merged matching config using hostname/originalHostname/user/localUser
      • This is mostly the same with how OpenSSH handles this
      • We go through all configs that have a Host ... or Match ... that matches our current context
      • We make sure our search context stays up-to-date (e.g. replace hostname once it's specified in a config)
      • We keep track of all directives in order, e.g. we keep track of every User ... and IdentityFile ...
      • We copy the first occurrence of several fields (if present) after parsing into a temporary overrides object
        • For now that's host, agent, tryKeyboard, hops, ... and a bunch of others, although not all are supported yet
      • At the end, we merge the overrides into the config we had before we started this whole SSH config process
    • If privateKeyPath is present, we read it and store it under privateKey
    • We prompt for any missing host, username and password fields (depending on certain factors)
      • E.g. password isn't prompted unless explicitly told to do so, by having password be the boolean true
    • If password is given, we disable agent (the underlying ssh2 library had issues dealing with both)
    • If passphrase is set to prompt, we do that if privateKey is set, otherwise warn user and exit
    • If there's no privateKey, no agent and no password, we prompt for a password anyway
• We now have the "actual" config, which will used to create the connection, render the connections tree view, ...
• At this point, the config is basically final and will be used to create the connection
  • While creating the connection, it might alter slight things if the server lacks support for something

This "overview" is a lot more complex/verbose than I expected, but it's good to have written down, including for myself

It's a very complex progress I have yet to simplify (where possible, at least) and make work better in regards to different config systems, e.g. OpenSSH config systems. I also have to work on adding support for a lot of directives and dealing with certain parsing/technical issues. For example, OpenSSH supports multiple IdentityFile directives which isn't supported by the extension (and the underlying ssh2 library) for now. There's also a need to match OpenSSH's behavior as close as possible, i.e. the interaction between all the different proxy-related directives, how defining one directive might interfere with another directive, when (not) to parse quotes or environment variables, ...

Currently I'm working on supporting more shells (e.g. fish) for mostly basic usage of the extension. Such an advanced feature as parsing OpenSSH configs is still something I want to do and am still working on, but I'm focusing on other things too.