more than 100 restarts in less than a minute

[xBiei]

trafficstars

Prerequisites

• [X] Write a descriptive title.
• [X] Make sure you are able to repro it on the latest version
• [X] Search the existing issues.

Steps to reproduce

• Stop sshd service "net stop sshd"
• Change sshd_config HostKey value
• Remove other keys
• Start sshd service "net start sshd"

Expected behavior

Run sshd successfully

Actual behavior

very fast unstoppable restarts

Error details

No response

Environment data

PSVersion                      7.2.5
PSEdition                      Core
GitCommitId                    7.2.5
OS                             Microsoft Windows 10.0.22000
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Version

V8.9.1.0p1-Beta

Visuals

it's actually waaayyy faster than what's showing in the gif but sharex couldn't even catch all frames. that doesn't look normal lol also I haven't done a restart which will probably fix this. connections to the server isn't working while in this state.

[xBiei]

actually I just restarted and it's the same state.

[xBiei]

this is what services manager shows. for like 500ms it goes to stopped status then running then goes back to starting again

[xBiei]

when I changed startup type to Disabled it stopped finally

[xBiei]

ok so finally after removing sshd_config it stopped the million restarts per second

[xBiei]

I checked the Event Viewer looks like it did 30,900 Errors in the last few minutes lmao

All 30K errors say the same thing without explanation to what caused the error Errors Description: The OpenSSH SSH Server service terminated unexpectedly. It has done this 22117 time(s). The following corrective action will be taken in 0 milliseconds: Restart the service.

[jborean93]

Try starting the sshd.exe as SYSTEM manually and seeing what error appears. You can use psexec.exe to start as SYSTEM and potentially add -d, -dd, -ddd to increase the verbosity of the logs as per https://github.com/PowerShell/Win32-OpenSSH/wiki/Troubleshooting-Steps.

[xBiei]

I'll check on that later I fixed it for now but I think I gotta reproduce it to get this bug fixed

[conquhare]

I encountered the same situation. I found that the possible reason was because of an error in the configuration file. I cut the bottom match content to the middle part of the file. After saving, restart the service and there will be an infinite loop restart phenomenon. Put it back in its original location and the service can be started normally.

[vthiebaut10]

@xBiel - Hi, I'm having issues with reproducing the issue. Sounds like it was an issue with the sshd_config file. Can you share the sshd_config file that caused the issue? Also, try starting sshd in debug mode manually, like @jborean93 recommended.

[xBiei]

@xBiel - Hi, I'm having issues with reproducing the issue. Sounds like it was an issue with the sshd_config file. Can you share the sshd_config file that caused the issue?

Also, try starting sshd in debug mode manually, like @jborean93 recommended.

I don't remember what I did exactly when that happened, all I remember is that I changed the sshd_config's HostKey value and restarted the service and it just never stopped restarting even after rebooting until I removed the sshd_config file.

I'll try reproducing the issue now.

[xBiei]

ok so I figured out why this is happening. I opened the default sshd_config file and edited the HostKey to match my private key file location which is __PROGRAMDATA__/ssh/id_rsa so my final sshd_config file looks like this:

HostKey __PROGRAMDATA__/ssh/id_rsa
AuthorizedKeysFile	.ssh/authorized_keys
Subsystem	sftp	sftp-server.exe
Match Group administrators
       AuthorizedKeysFile __PROGRAMDATA__/ssh/administrators_authorized_keys

then I restarted sshd service using the task manager services tab, then the issue started happening again. I'll try running in debug mode.

[xBiei]

Tried running on debug mode and it appears to be bad permissions error as seen below.

I think the best thing to do here is to make sshd handle this error differently, instead of existing immediately it should show an error window to the user to tell them that the HostKey file's permission needs to be changed to a more secure permissions (Ownership to SYSTEM, no other permissions).

But the weird thing is that I encountered this on my Arch Linux before and instead of restarting like that, it just exists and stops the service. Shouldn't windows be doing that too?

[xBiei]

The solution to fixing this issue is removing all permissions from the HostKey file and giving the ownership to SYSTEM. Everything should go back to normal. But this error should be shown to the user in a more obvious way instead of using debug mode and even using the terminal, I think an error window is the best and the most obvious way to tell the user.

[conquhare]

The root cause of unlimited restarts is that the openssh windows version of the program does not have error handling logic, and another secondary reason is that the action in the event of failure in the options configured during service installation is to restart the service.This service failure action can be found in services.Find the openssh service label in the msc service management interface to adjust.

---- Replied Message ---- | From | @.> | | Date | 10/18/2022 07:52 | | To | @.> | | Cc | @.@.> | | Subject | Re: [PowerShell/Win32-OpenSSH] more than 100 restarts in less than a minute (Issue #1970) |

The solution to fixing this issue is removing all permissions from the HostKey file and giving the ownership to SYSTEM. Everything should go back to normal. But this error should be shown to the user in a more obvious way instead of using debug mode and even using the terminal, I think an error window is the best and the most obvious way to tell the user.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[xBiei]

the openssh windows version of the program does not have error handling logic

Well, I think it should have at least an alert just to get rid of the confusion and the debugging time for the users. Just a suggestion.

[datdamnmachine]

I had this very same issue and wanted to update what worked for me. I checked the permissions of the host key files in the C:\ProgramData\ssh with this script:

# Specify the directory path
$directoryPath = "C:\ProgramData\ssh"

# Get a list of files that begin with "ssh_host_"
$files = Get-ChildItem -Path $directoryPath -Filter "ssh_host_*"

# Loop through each file and display ownership and permissions
foreach ($file in $files) {
    $filePath = $file.FullName
    $acl = Get-Acl -Path $filePath

    Write-Host "File: $($file.Name)"
    Write-Host "  Owner: $($acl.Owner)"
    Write-Host "  Permissions:"
    
    foreach ($ace in $acl.Access) {
        Write-Host "    $($ace.IdentityReference) - $($ace.FileSystemRights)"
    }

    Write-Host "-----------------------------------"
}

Everything looked fine, even though it showed the local Administrator permissions: For example:

Non-Working Machine:

File: ssh_host_dsa_key
  Owner: NT AUTHORITY\SYSTEM
  Permissions:
    NT AUTHORITY\SYSTEM - FullControl
    BUILTIN\Administrators - FullControl
-----------------------------------
File: ssh_host_dsa_key.pub
  Owner: NT AUTHORITY\SYSTEM
  Permissions:
    NT AUTHORITY\SYSTEM - FullControl
    BUILTIN\Administrators - FullControl
-----------------------------------
File: ssh_host_ecdsa_key
  Owner: NT AUTHORITY\SYSTEM
  Permissions:
    NT AUTHORITY\SYSTEM - FullControl
    BUILTIN\Administrators - FullControl
-----------------------------------
File: ssh_host_ecdsa_key.pub
  Owner: NT AUTHORITY\SYSTEM
  Permissions:
    NT AUTHORITY\SYSTEM - FullControl
    BUILTIN\Administrators - FullControl
-----------------------------------
File: ssh_host_ed25519_key
  Owner: NT AUTHORITY\SYSTEM
  Permissions:
    NT AUTHORITY\SYSTEM - FullControl
    BUILTIN\Administrators - FullControl
-----------------------------------
File: ssh_host_ed25519_key.pub
  Owner: NT AUTHORITY\SYSTEM
  Permissions:
    NT AUTHORITY\SYSTEM - FullControl
    BUILTIN\Administrators - FullControl
-----------------------------------
File: ssh_host_rsa_key
  Owner: NT AUTHORITY\SYSTEM
  Permissions:
    NT AUTHORITY\SYSTEM - FullControl
    BUILTIN\Administrators - FullControl
-----------------------------------
File: ssh_host_rsa_key.pub
  Owner: NT AUTHORITY\SYSTEM
  Permissions:
    NT AUTHORITY\SYSTEM - FullControl
    BUILTIN\Administrators - FullControl
-----------------------------------

Working Machine:

File: ssh_host_dsa_key
  Owner: NT AUTHORITY\SYSTEM
  Permissions:
    NT AUTHORITY\SYSTEM - FullControl
    BUILTIN\Administrators - FullControl
-----------------------------------
File: ssh_host_dsa_key.pub
  Owner: NT AUTHORITY\SYSTEM
  Permissions:
    NT AUTHORITY\SYSTEM - FullControl
    BUILTIN\Administrators - FullControl
-----------------------------------
File: ssh_host_ecdsa_key
  Owner: NT AUTHORITY\SYSTEM
  Permissions:
    NT AUTHORITY\SYSTEM - FullControl
    BUILTIN\Administrators - FullControl
-----------------------------------
File: ssh_host_ecdsa_key.pub
  Owner: NT AUTHORITY\SYSTEM
  Permissions:
    NT AUTHORITY\SYSTEM - FullControl
    BUILTIN\Administrators - FullControl
-----------------------------------
File: ssh_host_ed25519_key
  Owner: NT AUTHORITY\SYSTEM
  Permissions:
    NT AUTHORITY\SYSTEM - FullControl
    BUILTIN\Administrators - FullControl
-----------------------------------
File: ssh_host_ed25519_key.pub
  Owner: NT AUTHORITY\SYSTEM
  Permissions:
    NT AUTHORITY\SYSTEM - FullControl
    BUILTIN\Administrators - FullControl
-----------------------------------
File: ssh_host_rsa_key
  Owner: NT AUTHORITY\SYSTEM
  Permissions:
    NT AUTHORITY\SYSTEM - FullControl
    BUILTIN\Administrators - FullControl
-----------------------------------
File: ssh_host_rsa_key.pub
  Owner: NT AUTHORITY\SYSTEM
  Permissions:
    NT AUTHORITY\SYSTEM - FullControl
    BUILTIN\Administrators - FullControl
-----------------------------------

I'm assuming that is because it has to temporarily give itself permissions in order to be able to read the permissions. I also ran diagnostics commands such as these:

https://github.com/PowerShell/Win32-OpenSSH/wiki/Troubleshooting-Steps

.\psexec -s "C:\Program Files\OpenSSH\sshd.exe"  -ddd

.\psexec -s "C:\Program Files\OpenSSH\sshd.exe" -f "C:\ProgramData\ssh\sshd_config" -ddd

.\psexec -s "C:\Program Files\OpenSSH\sshd.exe" -f "C:\ProgramData\ssh\sshd_config" -h "C:\ProgramData\ssh\ssh_host_dsa_key" -ddd

.\psexec -s "C:\Program Files\OpenSSH\sshd.exe" -f "C:\ProgramData\ssh\sshd_config" -h "C:\ProgramData\ssh\ssh_host_rsa_key" -ddd

.\psexec -s "C:\Program Files\OpenSSH\sshd.exe" -f "C:\ProgramData\ssh\sshd_config" -h "C:\ProgramData\ssh\ssh_host_ecdsa_key" -ddd

.\psexec -s "C:\Program Files\OpenSSH\sshd.exe" -f "C:\ProgramData\ssh\sshd_config" -h "C:\ProgramData\ssh\ssh_host_ed25519_key" -ddd

They all worked and I was able to initiate a ssh client connection. One that didn't work, was the ssh_host_dsa_key. It would make the connection attempt but fail as the client can't match with the host key:

debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: (no match)
Unable to negotiate with 1.2.3.4 port 22: no matching host key type found. Their offer:

I decided to go scorched earth. I uninstalled, and made sure the following folders where deleted:

C:\Program Files\OpenSSH
C:\ProgramData\ssh

I then just reinstalled with the installer. This worked. I noticed that the ssh_host_dsa_key was not regenerated. I wonder if that key was the cause of the issues, even thought, when ran with psexec, it would still start the server.

I have some build template commands that regenerate the keys, based on some Linux steps, that I imported over to my Windows builds. I went ahead and removed those steps as, it appears, it isn't necessary.

I hope this helps someone else.