dbatools icon indicating copy to clipboard operation
dbatools copied to clipboard

Published 20 hours ago • verified publisher icon dataplat

Import module fails with concurrent update on non-concurrent collection exception

Open kborowinski opened this issue 2 years ago • 1 comments

Verified issue does not already exist?

Yes

What error did you receive?

PSMessageDetails      : 
Exception             : System.Management.Automation.MethodInvocationException: Exception calling "InvokeScript" with "4" argument(s): "Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct."
                         ---> System.Management.Automation.CmdletInvocationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
                         ---> System.Management.Automation.RuntimeException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
                         ---> System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
                           at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
                           at CallSite.Target(Closure, CallSite, Dictionary`2, Object, Object)
                           at System.Dynamic.UpdateDelegates.UpdateAndExecute3[T0,T1,T2,TRet](CallSite site, T0 arg0, T1 arg1, T2 arg2)
                           at System.Management.Automation.Interpreter.DynamicInstruction`4.Run(InterpretedFrame frame) in D:\DEVELOPMENT\PowerShellCore\src\System.Management.Automation\engine\interpreter\DynamicInstructions.Generated.cs:line 192
                           at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame) in D:\DEVELOPMENT\PowerShellCore\src\System.Management.Automation\engine\interpreter\ControlFlowInstructions.cs:line 358
                           --- End of inner exception stack trace ---
                           at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcContext, Exception exception) in D:\DEVELOPMENT\PowerShellCore\src\System.Management.Automation\engine\runtime\Operations\MiscOps.cs:line 1682
                           at System.Management.Automation.Interpreter.ActionCallInstruction`2.Run(InterpretedFrame frame) in D:\DEVELOPMENT\PowerShellCore\src\System.Management.Automation\engine\interpreter\CallInstruction.Generated.cs:line 504
                           at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame) in D:\DEVELOPMENT\PowerShellCore\src\System.Management.Automation\engine\interpreter\ControlFlowInstructions.cs:line 392
                           at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame) in D:\DEVELOPMENT\PowerShellCore\src\System.Management.Automation\engine\interpreter\ControlFlowInstructions.cs:line 358
                           at System.Management.Automation.Interpreter.Interpreter.Run(InterpretedFrame frame) in D:\DEVELOPMENT\PowerShellCore\src\System.Management.Automation\engine\interpreter\Interpreter.cs:line 101
                           at System.Management.Automation.Interpreter.LightLambda.RunVoid1[T0](T0 arg0) in D:\DEVELOPMENT\PowerShellCore\src\System.Management.Automation\engine\interpreter\LightLambda.Generated.cs:line 82
                           at System.Management.Automation.PSScriptCmdlet.RunClause(Action`1 clause, Object dollarUnderbar, Object inputToProcess) in D:\DEVELOPMENT\PowerShellCore\src\System.Management.Automation\engine\runtime\CompiledScriptBlock.cs:line 2407
                           at System.Management.Automation.PSScriptCmdlet.DoEndProcessing() in D:\DEVELOPMENT\PowerShellCore\src\System.Management.Automation\engine\runtime\CompiledScriptBlock.cs:line 2315
                           at System.Management.Automation.CommandProcessorBase.Complete() in D:\DEVELOPMENT\PowerShellCore\src\System.Management.Automation\engine\CommandProcessorBase.cs:line 596
                           --- End of inner exception stack trace ---
                           at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcContext, Exception exception) in D:\DEVELOPMENT\PowerShellCore\src\System.Management.Automation\engine\runtime\Operations\MiscOps.cs:line 1682
                           at System.Management.Automation.Interpreter.ActionCallInstruction`2.Run(InterpretedFrame frame) in D:\DEVELOPMENT\PowerShellCore\src\System.Management.Automation\engine\interpreter\CallInstruction.Generated.cs:line 504
                           at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame) in D:\DEVELOPMENT\PowerShellCore\src\System.Management.Automation\engine\interpreter\ControlFlowInstructions.cs:line 392
                           at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame) in D:\DEVELOPMENT\PowerShellCore\src\System.Management.Automation\engine\interpreter\ControlFlowInstructions.cs:line 358
                           at System.Management.Automation.Interpreter.Interpreter.Run(InterpretedFrame frame) in D:\DEVELOPMENT\PowerShellCore\src\System.Management.Automation\engine\interpreter\Interpreter.cs:line 101
                           at System.Management.Automation.Interpreter.LightLambda.RunVoid1[T0](T0 arg0) in D:\DEVELOPMENT\PowerShellCore\src\System.Management.Automation\engine\interpreter\LightLambda.Generated.cs:line 82
                           at System.Management.Automation.ScriptBlock.InvokeWithPipeImpl(ScriptBlockClauseToInvoke clauseToInvoke, Boolean createLocalScope, Dictionary`2 functionsToDefine, List`1 variablesToDefine, ErrorHandlingBehavior errorHandlingBehavior, Object dollarUnder, Object input, Object scriptThis, Pipe outputPipe, InvocationInfo invocationInfo, Object[] args) in D:\DEVELOPMENT\PowerShellCore\src\System.Management.Automation\engine\runtime\CompiledScriptBlock.cs:line 1218
                           at System.Management.Automation.ScriptBlock.InvokeWithPipe(Boolean useLocalScope, ErrorHandlingBehavior errorHandlingBehavior, Object dollarUnder, Object input, Object scriptThis, Pipe outputPipe, InvocationInfo invocationInfo, Boolean propagateAllExceptionsToTop, List`1 variablesToDefine, Dictionary`2 functionsToDefine, Object[] args) in D:\DEVELOPMENT\PowerShellCore\src\System.Management.Automation\engine\lang\scriptblock.cs:line 1043
                           at System.Management.Automation.ScriptBlock.DoInvokeReturnAsIs(Boolean useLocalScope, ErrorHandlingBehavior errorHandlingBehavior, Object dollarUnder, Object input, Object scriptThis, Object[] args) in D:\DEVELOPMENT\PowerShellCore\src\System.Management.Automation\engine\lang\scriptblock.cs:line 937
                           at System.Management.Automation.CommandInvocationIntrinsics.InvokeScript(ScriptBlock sb, Boolean useNewScope, PipelineResultTypes writeToPipeline, IList input, Object[] args) in D:\DEVELOPMENT\PowerShellCore\src\System.Management.Automation\engine\MshCmdlet.cs:line 864
                           at System.Management.Automation.CommandInvocationIntrinsics.InvokeScript(Boolean useLocalScope, ScriptBlock scriptBlock, IList input, Object[] args) in D:\DEVELOPMENT\PowerShellCore\src\System.Management.Automation\engine\MshCmdlet.cs:line 772
                           at CallSite.Target(Closure, CallSite, Object, Boolean, Object, Object, Object)
                           --- End of inner exception stack trace ---
                           at System.Management.Automation.ExceptionHandlingOps.ConvertToMethodInvocationException(Exception exception, Type typeToThrow, String methodName, Int32 numArgs, MemberInfo memberInfo) in D:\DEVELOPMENT\PowerShellCore\src\System.Management.Automation\engine\runtime\Operations\MiscOps.cs:line 2095
                           at CallSite.Target(Closure, CallSite, Object, Boolean, Object, Object, Object)
                           at System.Dynamic.UpdateDelegates.UpdateAndExecute5[T0,T1,T2,T3,T4,TRet](CallSite site, T0 arg0, T1 arg1, T2 arg2, T3 arg3, T4 arg4)
                           at System.Management.Automation.Interpreter.DynamicInstruction`6.Run(InterpretedFrame frame) in D:\DEVELOPMENT\PowerShellCore\src\System.Management.Automation\engine\interpreter\DynamicInstructions.Generated.cs:line 244
                           at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame) in D:\DEVELOPMENT\PowerShellCore\src\System.Management.Automation\engine\interpreter\ControlFlowInstructions.cs:line 358
TargetObject          : 
CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
FullyQualifiedErrorId : CmdletInvocationException
ErrorDetails          : 
InvocationInfo        : System.Management.Automation.InvocationInfo
ScriptStackTrace      : at <ScriptBlock>, C:\Program Files\WindowsPowerShell\Modules\dbatools\1.1.95\internal\scripts\dbatools-maintenance.ps1: line 3
                        at <ScriptBlock>, C:\Program Files\WindowsPowerShell\Modules\dbatools\1.1.95\dbatools.psm1: line 304
                        at <ScriptBlock>, <No file>: line 1
PipelineIterationInfo : {}

Steps to Reproduce

I have infrastructure testing module that requires dbatools and thus imports it in module manifest: image

The problem is that once a while (like 1 in 10 times) I get following dbatools module import exception: dbatools_import_exception

If I'm not mistaken it looks like the non-concurrent collection is modified concurrently during module initialization.

Are you running the latest release?

Yes

Other details or mentions

No response

What PowerShell host was used when producing this error

PowerShell Core (pwsh.exe)

PowerShell Host Version

Name                           Value
----                           -----
PSVersion                      7.3.0-preview.3
PSEdition                      Core
GitCommitId                    7.3.0-preview.3-78-gea0ab84eb2e0e17f9ac28708737cf9ae8744dcae
OS                             Microsoft Windows 10.0.19044
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0, 5.0, 5.1.10032.0, 6.0.0, 6.1.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

SQL Server Edition and Build number

Microsoft SQL Server 2016 (SP3) (KB5003279) - 13.0.6300.2 (X64) Aug 7 2021 01:20:37 Copyright (c) Microsoft Corporation Enterprise Edition (64-bit) on Windows Server 2012 R2 Datacenter 6.3 <X64> (Build 9600: )

.NET Framework Version

.NET 7.0.0-preview.4.22229.4

kborowinski avatar May 18 '22 07:05 kborowinski

It just happened again, this time on PowerShell Core 7.2.4: pwsh_7_2_4_dbatools_exception

Exception:


PSMessageDetails      : 
Exception             : System.Management.Automation.CommandNotFoundException: The 'test-KBInfrastructure' command was found in the module 'KBInfrastructureTesting', but the module could not be loaded. For more information, run 'Import-Module KBInfrastructureTesting'.
                           at System.Management.Automation.CommandDiscovery.TryModuleAutoDiscovery(String commandName, ExecutionContext context, String originalCommandName, CommandOrigin commandOrigin, SearchResolutionOptions searchResolutionOptions, CommandTypes commandTypes, Exception& lastError)
                           at System.Management.Automation.CommandDiscovery.LookupCommandInfo(String commandName, CommandTypes commandTypes, SearchResolutionOptions searchResolutionOptions, CommandOrigin commandOrigin, ExecutionContext context)
                           at System.Management.Automation.ExecutionContext.CreateCommand(String command, Boolean dotSource)
                           at System.Management.Automation.PipelineOps.AddCommand(PipelineProcessor pipe, CommandParameterInternal[] commandElements, CommandBaseAst commandBaseAst, CommandRedirection[] redirections, ExecutionContext context)
                           at System.Management.Automation.PipelineOps.InvokePipeline(Object input, Boolean ignoreInput, CommandParameterInternal[][] pipeElements, CommandBaseAst[] pipeElementAsts, CommandRedirection[][] commandRedirections, FunctionContext funcContext)
                           at System.Management.Automation.Interpreter.ActionCallInstruction`6.Run(InterpretedFrame frame)
                           at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
TargetObject          : test-KBInfrastructure
CategoryInfo          : ObjectNotFound: (test-KBInfrastructure:String) [], CommandNotFoundException
FullyQualifiedErrorId : CouldNotAutoloadMatchingModule
ErrorDetails          : 
InvocationInfo        : System.Management.Automation.InvocationInfo
ScriptStackTrace      : at <ScriptBlock>, <No file>: line 1
PipelineIterationInfo : {}

PSMessageDetails      : 
Exception             : System.Exception: Failed to start runspace: dbatools-maintenance
TargetObject          : dbatools-maintenance
CategoryInfo          : NotSpecified: (dbatools-maintenance:String) [Write-Error], Exception
FullyQualifiedErrorId : dbatools_Start-DbaRunspace,Stop-Function
ErrorDetails          : Failed to start runspace: dbatools-maintenance
InvocationInfo        : System.Management.Automation.InvocationInfo
ScriptStackTrace      : at Stop-Function, C:\Program Files\WindowsPowerShell\Modules\dbatools\1.1.95\allcommands.ps1: line 147768
                        at Start-DbaRunspace<Process>, C:\Program Files\WindowsPowerShell\Modules\dbatools\1.1.95\allcommands.ps1: line 149225
                        at <ScriptBlock>, C:\Program Files\WindowsPowerShell\Modules\dbatools\1.1.95\internal\scripts\dbatools-maintenance.ps1: line 40
                        at <ScriptBlock>, C:\Program Files\WindowsPowerShell\Modules\dbatools\1.1.95\dbatools.psm1: line 304
                        at <ScriptBlock>, <No file>: line 1
PipelineIterationInfo : {0, 1}

PSMessageDetails      : 
Exception             : System.Management.Automation.MethodInvocationException: Exception calling "Start" with "0" argument(s): "The state of the current PowerShell instance is not valid for this operation."
                         ---> System.Management.Automation.InvalidPowerShellStateException: The state of the current PowerShell instance is not valid for this operation.
                           at System.Management.Automation.PowerShell.AssertChangesAreAccepted()
                           at System.Management.Automation.PowerShell.AddScript(String script)
                           at Sqlcollaborative.Dbatools.Runspace.RunspaceContainer.Start() in F:\Code\Github\dbatools\bin\projects\dbatools\dbatools\Runspace\RunspaceContainer.cs:line 63
                           at CallSite.Target(Closure , CallSite , Object )
                           --- End of inner exception stack trace ---
                           at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcContext, Exception exception)
                           at System.Management.Automation.Interpreter.ActionCallInstruction`2.Run(InterpretedFrame frame)
                           at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
                           at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
TargetObject          : 
CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
FullyQualifiedErrorId : InvalidPowerShellStateException
ErrorDetails          : 
InvocationInfo        : System.Management.Automation.InvocationInfo
ScriptStackTrace      : at Start-DbaRunspace<Process>, C:\Program Files\WindowsPowerShell\Modules\dbatools\1.1.95\allcommands.ps1: line 149223
                        at <ScriptBlock>, C:\Program Files\WindowsPowerShell\Modules\dbatools\1.1.95\internal\scripts\dbatools-maintenance.ps1: line 40
                        at <ScriptBlock>, C:\Program Files\WindowsPowerShell\Modules\dbatools\1.1.95\dbatools.psm1: line 304
                        at <ScriptBlock>, <No file>: line 1
PipelineIterationInfo : {}

PSMessageDetails      : 
Exception             : System.Exception: Failed to start runspace: dbatools-teppasynccache
TargetObject          : dbatools-teppasynccache
CategoryInfo          : NotSpecified: (dbatools-teppasynccache:String) [Write-Error], Exception
FullyQualifiedErrorId : dbatools_Start-DbaRunspace,Stop-Function
ErrorDetails          : Failed to start runspace: dbatools-teppasynccache
InvocationInfo        : System.Management.Automation.InvocationInfo
ScriptStackTrace      : at Stop-Function, C:\Program Files\WindowsPowerShell\Modules\dbatools\1.1.95\allcommands.ps1: line 147768
                        at Start-DbaRunspace<Process>, C:\Program Files\WindowsPowerShell\Modules\dbatools\1.1.95\allcommands.ps1: line 149225
                        at <ScriptBlock>, C:\Program Files\WindowsPowerShell\Modules\dbatools\1.1.95\internal\scripts\updateTeppAsync.ps1: line 93
                        at <ScriptBlock>, C:\Program Files\WindowsPowerShell\Modules\dbatools\1.1.95\dbatools.psm1: line 300
                        at <ScriptBlock>, <No file>: line 1
PipelineIterationInfo : {0, 1}

PSMessageDetails      : 
Exception             : System.Management.Automation.MethodInvocationException: Exception calling "Start" with "0" argument(s): "The state of the current PowerShell instance is not valid for this operation."
                         ---> System.Management.Automation.InvalidPowerShellStateException: The state of the current PowerShell instance is not valid for this operation.
                           at System.Management.Automation.PowerShell.AssertChangesAreAccepted()
                           at System.Management.Automation.PowerShell.AddScript(String script)
                           at Sqlcollaborative.Dbatools.Runspace.RunspaceContainer.Start() in F:\Code\Github\dbatools\bin\projects\dbatools\dbatools\Runspace\RunspaceContainer.cs:line 63
                           at CallSite.Target(Closure , CallSite , Object )
                           --- End of inner exception stack trace ---
                           at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcContext, Exception exception)
                           at System.Management.Automation.Interpreter.ActionCallInstruction`2.Run(InterpretedFrame frame)
                           at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
                           at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
TargetObject          : 
CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
FullyQualifiedErrorId : InvalidPowerShellStateException
ErrorDetails          : 
InvocationInfo        : System.Management.Automation.InvocationInfo
ScriptStackTrace      : at Start-DbaRunspace<Process>, C:\Program Files\WindowsPowerShell\Modules\dbatools\1.1.95\allcommands.ps1: line 149223
                        at <ScriptBlock>, C:\Program Files\WindowsPowerShell\Modules\dbatools\1.1.95\internal\scripts\updateTeppAsync.ps1: line 93
                        at <ScriptBlock>, C:\Program Files\WindowsPowerShell\Modules\dbatools\1.1.95\dbatools.psm1: line 300
                        at <ScriptBlock>, <No file>: line 1
PipelineIterationInfo : {}

PSMessageDetails      : 
Exception             : System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
                           at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
                           at CallSite.Target(Closure , CallSite , Dictionary`2 , Object , Object )
                           at System.Dynamic.UpdateDelegates.UpdateAndExecute3[T0,T1,T2,TRet](CallSite site, T0 arg0, T1 arg1, T2 arg2)
                           at System.Management.Automation.Interpreter.DynamicInstruction`4.Run(InterpretedFrame frame)
                           at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
TargetObject          : 
CategoryInfo          : OperationStopped: (:) [], InvalidOperationException
FullyQualifiedErrorId : System.InvalidOperationException
ErrorDetails          : 
InvocationInfo        : System.Management.Automation.InvocationInfo
ScriptStackTrace      : at Register-DbaRunspace, C:\Program Files\WindowsPowerShell\Modules\dbatools\1.1.95\allcommands.ps1: line 149173
                        at <ScriptBlock>, C:\Program Files\WindowsPowerShell\Modules\dbatools\1.1.95\internal\scripts\logfilescript.ps1: line 134
                        at <ScriptBlock>, C:\Program Files\WindowsPowerShell\Modules\dbatools\1.1.95\dbatools.psm1: line 296
                        at <ScriptBlock>, <No file>: line 1
PipelineIterationInfo : {}

kborowinski avatar May 18 '22 14:05 kborowinski

@kborowinski - is this still the case with the latest version of dbatools?

@potatoqualitee - can you have a look at this?

andreasjordan avatar Oct 24 '22 13:10 andreasjordan

oh wow, i never even noticed this issue. this is C# and multithreading which I can only handle when there are no issues 💀 perhaps @FriedrichWeinmann could offer some insight.

This may get better in Core when I release dbatools 2.0, as it's based on .NET 6

potatoqualitee avatar Oct 24 '22 14:10 potatoqualitee

can you tell us more about the KB modules, @kborowinski ? What kind of DLLs do they import?

potatoqualitee avatar Oct 25 '22 07:10 potatoqualitee

I'll do guys but next week. Now I'm sailing in Croatia ⛵

kborowinski avatar Oct 25 '22 07:10 kborowinski

I think that the problem is related to a bug in CompletionPredictor module, that loads dbatools module into own separate Runspace causing this issue. It boils down to a fact the custom types will be initialized twice. Unfortunately since that time I've refactored my code to use SQLServer module in place of dbatools so I did not test the recent module versions anymore.

Ps. For anyone interested, as a workaround I do not use HistoryAndPlugin option in PSReadLine and I do not import CompletionPredictor module anymore.

kborowinski avatar Oct 30 '22 20:10 kborowinski

Hi @kborowinski thanks for all the details. I will close this now but can reopen if needed.

andreasjordan avatar Nov 01 '22 07:11 andreasjordan