dbatools

dbatools copied to clipboard

Verified issue does not already exist?

No

What error did you receive?

$UpdateDbaInstanceSplat = @{ Credential = $credential ComputerName = "$Target_ServerName" Path = "\NetworkShare\AutoPatch" ExtractPath = "E:\TempUpdate" Version = "CU15" } Update-DbaInstance @UpdateDbaInstanceSplat -confirm:$true -Verbose -Restart
VERBOSE: [12:03:50][Invoke-Program] Starting process [E:\TempUpdate\dbatools_KB5008996_Extract_8088ed3d42cc4e76a23bc78d0d439423\setup.exe] with arguments [/quiet /allinstances /IAcceptSQLServerLicenseTerms] on MyComputer through Credssp protocol HANG OCCURS but setup.exe is finished - not getting anything back from Invoke-Program? Is is at Invoke_Program or remove temporary files? I restart the Remote Server manually, and get this >>>>

WARNING: [12:09:34][Invoke-DbaAdvancedUpdate] Update failed with exit code Confirm Are you sure you want to perform this action? Performing the operation "Set variable" on target "Name: __dbatools_interrupt_function_78Q9VPrM6999g6zo24Qn83m09XF56InEn4hFrA8Fwhu5xJrs6r Value: True". Y ComputerName : MyComputer MajorVersion : 2019 TargetLevel : RTMCU15 KB : 5008996 Successful : False Restarted : False InstanceName : Installer : \NetworkShare\AutoPatch\SQLServer2019-KB5008996-CU15-x64.exe Notes : {Update failed with exit code }

WARNING: [12:09:39][Update-DbaInstance] Update failed: Update failed with exit code VERBOSE: [12:10:02][Invoke-DbaAdvancedUpdate] Failed to cleanup temp folder on computer MyComputer: Connecting to remote server MyComputer failed with the following error message : The client cannot connect to the destination specified in the request. Verify that the service on the destination is running and is accepting requests. Consult the logs and documentation for the WS-Management service running on the destination, most commonly IIS or WinRM. If the destination is the WinRM service, run the following command on the destination to analyze and configure the WinRM service: "winrm quickconfig". For more information, see the about_Remote_Troubleshooting Help topic.

My NOTE:The temporary Files do not get deleted, but the patch was successful - even though it Says FALSE

Steps to Reproduce

$UpdateDbaInstanceSplat = @{ Credential = $credential ComputerName = "$Target_ServerName" Path = "\NetworkShare\AutoPatch" ExtractPath = "E:\TempUpdate" Version = "CU15" } Update-DbaInstance @UpdateDbaInstanceSplat -confirm:$true -Verbose -Restart

Are you running the latest release?

Yes

Other details or mentions

This is happening every time on every server. Thank you for providing anything that could help in figuring out where the hang is happening.

What PowerShell host was used when producing this error

Windows PowerShell ISE (powershell_ise.exe)

PowerShell Host Version

Name Value

---

PSVersion 5.1.14393.4583
PSEdition Desktop
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0...}
BuildVersion 10.0.14393.4583
CLRVersion 4.0.30319.42000
WSManStackVersion 3.0
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1

SQL Server Edition and Build number

15.0.4198

.NET Framework Version

.NET Framework 4.7.3920.0

Are you able to Enter-PSSession $Target_ServerName successfully from your localhost? Or are you running the Update-DbaInstance from the box that SQL Server is sitting on ( e.g. Install-Module dbatools lives on the server)?

This is executed from separate server than the target. My NOTE:The temporary Files do not get deleted, but the patch was successful - even though it Says FALSE. I have not investigated Enter-PSSession.
I did open a ticket with Microsoft Support and have since updated to 1.1.83 dbatools and my experience with Update-DbaInstance is now different - something is changed, and we havent gotten a hang either.

I am at 1.1.88 Now, for a short time (2 weeks or so) there were no HANGS. Now there are new patches to install and I am getting the hangs back. It is right inbetween the bootstrap log and the REMOVE TEMP FILES. Wish there was some pointers to how to figure out why it is hanging.

VERBOSE: [11:35:13][Find-SqlInstanceUpdate] Using filter [SQLServer2019*-KB5011644-x64.exe] to check for updates in \NetworkShareSQLServer\Software\AutoPatch Confirm Are you sure you want to perform this action? Performing the operation "Update 2019 to RTMCU16 (KB5011644)" on target "Target.SQLServer.computer". &Yes Yes to &All &No No to A&ll &Suspend A VERBOSE: [11:35:15][Get-DbaCmObject] Configuration loaded | Cache disabled: False VERBOSE: [11:35:15][Get-DbaCmObject] [Target.SQLServer.computer] Retrieving Management Information VERBOSE: [11:35:15][Get-DbaCmObject] [Target.SQLServer.computer] Accessing computer using Cim over DCOM VERBOSE: [11:35:15][Get-DbaCmObject] [Target.SQLServer.computer] Accessing computer using Cim over DCOM - Success VERBOSE: [11:35:15][Invoke-DbaAdvancedUpdate] Extracting \NetworkShareSQLServer\Software\AutoPatch\SQLServer2019-KB5011644-CU16-x64.exe to E:\TempUpdate\dbatools_KB5011644_Extract_a995337b249547b383d07be1a10b30a1 VERBOSE: [11:35:15][Invoke-Program] Starting process [\NetworkShareSQLServer\Software\AutoPatch\SQLServer2019-KB5011644-CU16-x64.exe] with arguments [/x:"E:\TempUpdate\dbatools_KB5011644_Extract_a995337b249547b383d07be1a10b30a1" /quiet] on Target.SQLServer.computer through Credssp protocol VERBOSE: [11:35:42][Invoke-DbaAdvancedUpdate] Starting installation from E:\TempUpdate\dbatools_KB5011644_Extract_a995337b249547b383d07be1a10b30a1 VERBOSE: [11:35:42][Invoke-Program] Starting process [E:\TempUpdate\dbatools_KB5011644_Extract_a995337b249547b383d07be1a10b30a1\setup.exe] with arguments [/quiet /allinstances /IAcceptSQLServerLicenseTerms] on Target.SQLServer.computer through Credssp protocol

I KNOW the setup.exe is complete - I verified the bootstrap - Reboot is needed so I manually REBOOT the target server "Target.SQLServer.computer"

WARNING: [11:45:47][Invoke-DbaAdvancedUpdate] Update failed with exit code Confirm Are you sure you want to perform this action? Performing the operation "Set variable" on target "Name: __dbatools_interrupt_function_78Q9VPrM6999g6zo24Qn83m09XF56InEn4hFrA8Fwhu5xJrs6r Value: True". &Yes Yes to &All &No No to A&ll &Suspend Y

The script picks up with removing the E:\TempUpdate\dbatools_KB5011644_Extract_a995337b249547b383d07be1a10b30a1 and finishes.

Not sure. One could grab procmon from the sysinternals toolset. It can be installed with choco install sysinternals using Chocolatey package manager. Or you can grab it in a zip archive online. You can filter on the powershell and setup process to see if something is hung or if there is some registry key or file being written to. Maybe there is some Microsoft telemetry going on hanging it up.

What can I do to see if I am hanging on invoke-program $ps.WaitForExit()?

Uh. You could run the function in debug mode in VS Code or in the shell with Set-PSBreakpoint.

Now I have a client with this problem as well. It's a two-node-cluster with two instances (2017 and 2019) and today we patched both instances on both nodes (three times the passive side, one time the active side). Every time the same problem: Last line that was running is most likely $updateResult = Invoke-Program @execParams -Path "$spExtractPath\setup.exe" -ArgumentList $programArgumentList -WorkingDirectory $spExtractPath -Fallback. Updated dbatools yesterday, so it's current. I can not reproduce in my lab.

@nvarscar Do you have any idea?

Some test code to test just Invoke-Program:

$dbatools_enabledebug = $true
$moduleBase = Get-Module -Name dbatools -ListAvailable | Select-Object -First 1 -ExpandProperty ModuleBase
Import-Module "$moduleBase\dbatools.psm1"

$cred = Get-Credential
$params = @{
    Path            = "cmd.exe"
    ArgumentList    = '/c', '"timeout /T 300 && exit 2"'
    ComputerName    = 'sql01'
    Authentication  = 'Credssp'
    Credential      = $cred
    ErrorAction     = 'Stop'
    Fallback        = $true
    EnableException = $true 
}
Invoke-Program @params

Maybe that helps to reproduce the problem...

I reduced the number of parameters in Update-DbaInstance. Removed -Type -ExtractPath. I am not getting occasional hang condition between the patch completing and the removal of temporary files. Still testing but I am optimistic enough to let you know.

After removing -ExtractPath parameter we have had over 100 successes and no hangs. Although the algorithm for default on extractpath is not my preferred choice the hang issue has been eliminated.

Thanks for the report @salvbing -- @andreasjordan could salvbing's feedback help fix a bug?

Will have a look at that in the next days.

Some thoughts for documentation:

About the temp-files not beeing removed: This is most likely because it is tried while the server restarts and the WinRM service is not ready. I would not investigate this further.

If ExtractPath is not used, this code is used: $chosenDrive = (Get-DbaDiskSpace -ComputerName $computer -Credential $Credential -EnableException:$true | Sort-Object -Property Free -Descending | Select-Object -First 1).Name. If ExtractPath is used, then $chosenDrive = $ExtractPath. The actual path is build like this: $spExtractPath = $chosenDrive.TrimEnd('\') + "\dbatools_KB$($currentAction.KB)_Extract_$([guid]::NewGuid().Guid.Replace('-',''))".

So if E: is the drive with the most free space, then using ExtractPath with "E:" should make no difference in the used path. maybe @salvbing can try to test this?

The main difference here is that without ExtractPath, the used path is just one subfolder under the root directory of the drive, but in the failing code, there is an additional level. I still don't have an idea why this should make a difference.

I still can not reproduce the issue, and I currently have no further ideas.

Thanks for looking so deeply, @andreasjordan 🙇🏼

Will close this now.