PoshRSJob
PoshRSJob copied to clipboard
Published
20 hours ago •
proxb
proxb
Start-RSJob error when Collection in argument list is modified before all jobs have started.
I ran into a issue in Start-RSJobs module.
Collection was modified; enumeration operation may not execute. At E:\Scripts\MultiThread-RSJobs\PoshRSJob\Public\Start-RSJob.ps1:381 char:25 + ,$ArgumentList | ForEach { + ~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : OperationStopped: (:) [], InvalidOperationException + FullyQualifiedErrorId : System.InvalidOperationException
When the number of files in $targetfiles is fairly small 350ish the issue doesn't happen. It seems to occur when a large number of jobs are started and some of the first jobs are running and modifying the $SyncArrayLogOutput before all the other jobs have started. Here is the code I'm working on below.
$ErrorActionPreference = "Stop"
$path = $PSScriptRoot
# Import the RSJobs module.
Import-Module $path\PoshRSJob\PoshRSJob.psm1
#import AlphaFS for long path support.
Import-Module -Name "E:\Scripts\Modules\AlphaFS.dll"
$LogFile = $Path+"\Log.csv"
$LogFileErrors = $Path+"\ErrorLog.csv"
#region Functions
#Function to provide a more robust Synchronized variable locking and unlocking.
function Get-AlphaFSChildItem {
<#
.SYNOPSIS
Gets the files and folders in a file system drive and can handle Folder and File paths that are greater than 260 characters.
.DESCRIPTION
This command is similar to Get-ChildItem except it takes advantage of the AlphaFS(.NET) module in order to get around the 260 character path limit.
***************************************************************
The AlphaFS.dll must be downloaded: http://alphafs.alphaleonis.com/index.html
Then imported: PS C:\> Import-Module -Name "%FilePath%/AlphaFS.dll"
Information can be found: https://github.com/alphaleonis/AlphaFS/wiki/PowerShell
***************************************************************
.PARAMETER Path
.EXAMPLE
Get-AlphaFSChildItem
Description
-----------
This command gets the files and subdirectories in the current directory. If the current directory does not have
child items, the command does not return any results.
.EXAMPLE
Get-AlphaFSChildItem -path C:\ps-test
Description
-----------
This command gets the child items in the C:\ps-test directory
.EXAMPLE
Get-AlphaFSChildItem -path C:\ps-test -recurse
Description
-----------
This command gets the system files and folders in the specified directory and its subdirectories.
.EXAMPLE
Get-AlphaFSChildItem -path C:\ps-test -SearchPattern "*.txt"
Description
-----------
This command gets the system files that end in "*.txt"
.LINK
http://alphafs.alphaleonis.com/index.html
http://alphafs.alphaleonis.com/doc/2.0/index.html
https://github.com/alphaleonis/AlphaFS/wiki/PowerShell
#>
param(
[Parameter(
ValueFromPipeline=$True,
ValueFromPipelineByPropertyName=$True
)
][string]$Path = (Get-Location).Path,
[string]$SearchPattern = '*',
[array]$Include,
[array]$Exclude,
[Switch]$Recurse
)
$Error.Clear()
if(!(Get-Module -name AlphaFS -ErrorAction SilentlyContinue)){
Import-Module -name AlphaFS
}
if($Error){
return
}
$items = @()
$FileSystemEntryInfo = @()
if($recurse){
$SearchOption = 'AllDirectories'
}
else{
$SearchOption = 'TopDirectoryOnly'
}
# Get all folders
$array = @()
Try {
$array = [Alphaleonis.Win32.Filesystem.Directory]::EnumerateDirectories($Path,$SearchPattern,[System.IO.SearchOption]::$SearchOption)
Foreach ($file in $array) { $items += $file }
}
Catch [System.UnauthorizedAccessException] {
# Report exception.
Write-Error $_.Exception.Message
}
# Get all files
Try {
$array = [Alphaleonis.Win32.Filesystem.Directory]::EnumerateFiles($Path,$SearchPattern,[System.IO.SearchOption]::$SearchOption)
Foreach ($file in $array) { $items += $file }
}
Catch [System.UnauthorizedAccessException] {
# Report exception.
Write-Error $_.Exception.Message
}
foreach($item in $items){
$FileSystemEntryInfo += [Alphaleonis.Win32.Filesystem.File]::GetFileSystemEntryInfo($item)
}
if($Include){
$IncludedItems = @()
foreach($inc in $Include){
$IncludedItems += $FileSystemEntryInfo | Where-Object {$_.FullPath -like "$inc"}
}
$FileSystemEntryInfo = $IncludedItems
}
if($Exclude){
foreach($Exc in $Exclude){
$FileSystemEntryInfo = $FileSystemEntryInfo | Where-Object {$_.FullPath -notlike "$Exc"}
}
}
return $FileSystemEntryInfo | Select @{N='FullName'; E={$_.FullPath}},@{N='LongFullName'; E={$_.LongFullPath}},*
}
Function Write-Log {
[cmdletbinding()]
Param ([Parameter(ValueFromPipeline)]
[string[]]$MessageList
)
process
{
foreach ($Message in $MessageList) {
add-content $LogFile $Message
}
}
}
Function Write-ErrorLog {
[cmdletbinding()]
Param ([Parameter(ValueFromPipeline)]
[string[]]$MessageList
)
process
{
foreach ($Message in $MessageList) {
add-content $LogFileErrors $Message
}
}
}
function Get-AlphaFSHexString {
param(
[Parameter(Mandatory=$True,Position=1)]
[string]$Path = '',
[Parameter(Mandatory=$True,Position=2)]
[int]$Bytes = ''
)
$Signatures = @(
'0DE0AD0BE0EF040E70490880B202C01D027020010680D40C10B304D0C803A09F0DE0580C0C90310B0880C408D0AC0240F00C509401F03308C010',
'0DE0AD0BE0EF040920CB03C0320B80DA0BD0C90FF0CC01F050EC0E0580E10D60C80D307704F0C20E90C20C803E0570940D908B053043065081034',
'0DE0AD0BE0EF040DD0600450450360B106C0610CC0370790350B606707803607B02D0720670E60640330F80D70C90980710C7010CD0760FA0EE090',
'0DE0AD0BE0EF0409F08309F0FA02D0AE0CD06A06D0840F20920270440E707D020820FF05A082040130CF0920C06301701C061027089080E908F',
'0DE0AD0BE0EF040E707905808D0F009C0630E50650A30E00F00F10310707E08A0103103A0C50B40FD07003B08408202305B0F30DD0A703B0C509E',
'0DE0AD0BE0EF040FB0710660B20620E60C6062053060B20800970120AE0E50902105604F0B40300D30A80890490A00130DB04F0650C80410960EA',
'0DE0AD0BE0EF040FF0E00970F103E0B509105001C06409001B03309F0BD0D60D20540180F7083010560B0020AE010EB0FE05101E03501409F0B3',
'0DE0AD0BE0EF040D50270B80002B07609D04F0310550960820C308D0CF0170C50290DA0C0BD0410B909B0EC08A0810FC0DE07F01308D0D80A80D8',
'0DE0AD0BE0EF040DD0960C90CD0BA0BC0B102E03004D000A30F10700270D002F0770F10AD0FB0CF031084090A60A10DE0490EC0D10560AD0670EA',
'0DE0AD0BE0EF040B50330990490120B80BB0B707D0620D0C80C30580E208A08002109F0A901A0580D0500BE05807D0BA0DF02208905D000980FA',
'0DE0AD0BE0EF04080980F60FF0FC0160710906A01201508E0660640F609F0620350470301802908A05A02F0D50750A50C80C60F90B60E30E08C',
'0DE0AD0BE0EF0409B09C09A04604501C04B0440F902E09E0C20D00EB049080180A405903809507701A05C09B0A00F50FD0EF0FD0A604E016048068'
)
if ($HeaderHexString) { Remove-Variable HeaderHexString }
Try {
$File= [Alphaleonis.Win32.Filesystem.File]::OpenRead($Path)
for ($i = 0; $i -lt $Bytes; $i++)
{
if ($byte) { Remove-Variable byte }
$byte = $($File.ReadByte())
# if ( ( '{0:X}' -f $byte ).Length -eq 1 ) {
$HeaderHexString += '0{0:X}' -f $byte
# }
# else {
# $HeaderHexString += '{0:X}' -f $byte
# }
}
$File.Close()
}
Catch {
# Report exception.
}
$HeaderHexString
}
Get-RSJob | Remove-RSJob -force
#endregion
#Remove any stale jobs
Get-RSJob | Remove-RSJob
$SyncArrayLogOutput = [System.Collections.ArrayList]::Synchronized(@{}) ; $SyncArrayLogOutput.Clear()
$ScriptBlock = {
Param($SyncArrayLog)
$TD = get-Date -f "yyyy:MM:dd:HH:mm:ss.fff"
$ExecuteTime = $(Measure-Command {
if ($Header) { Remove-Variable Header }
$header = Get-AlphaFSHexString -path $_.LongFullName -Bytes 40
}).Milliseconds
[System.Threading.Monitor]::Enter($SyncArrayLog.SyncRoot)
$SyncArrayLog.add("$td,$($_.LongFullName),$ExecuteTime")
[System.Threading.Monitor]::Exit($SyncArrayLog.SyncRoot)
return $header
}
#$TargetFiles = get-AlphaFSchilditem -Path $TargetPath -Recurse
$TargetFiles = get-AlphaFSchilditem -Path E:\EncryptedArchive\twhite -Recurse
$TargetFiles | Start-RSjob -ScriptBlock $ScriptBlock -ArgumentList $SyncArrayLogOutput -FunctionsToLoad Get-AlphaFSHexString -ModulesToImport "E:\Scripts\Modules\AlphaFS.dll"
do {
sleep 1
$failed =Get-RSJob -State Failed
$failed | Write-ErrorLog
$failed | Remove-RSJob -force
Get-RSJob -State Completed | Remove-RSJob
write-host "Checking"
$temp = New-Object System.Collections.ArrayList($null)
[System.Threading.Monitor]::Enter($SyncArrayLogOutput.SyncRoot)
$temp.AddRange($SyncArrayLogOutput)
$SyncArrayLogOutput.Clear()
[System.Threading.Monitor]::Exit($SyncArrayLogOutput.SyncRoot)
write-host "-----------------------------------------------------------------"
$temp | Write-Log
$temp
}while ($($(Get-RSJob).length) -gt 0)
Thanks for providing the code to duplicate this issue. I'll take a look at it and see what I can find.
Just out of curiosity and I know this is many many months late, but I am curious if anything has changed with the recent updates that I have pushed out.
I do have this issue too with version 1.5.7.6.
It seems caused by [System.Collections.ArrayList]::Synchronized as it doesn't occur with a [hashtable]::Synchronized
Weird, did you use the same code as OP or did you use something different? If different, can you post it here?
@Fxbouffant @kwslavens Can you try again with the latest build and let me know if you still see issues?
@codykonior Just to make sure, the same error as this:
Collection was modified; enumeration operation may not execute.
At E:\Scripts\MultiThread-RSJobs\PoshRSJob\Public\Start-RSJob.ps1:381 char:25
+ ,$ArgumentList | ForEach {
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [], InvalidOperationException
+ FullyQualifiedErrorId : System.InvalidOperationException
lines from 241:
If ($PSBoundParameters.ContainsKey('ArgumentList')) {
If (@($ArgumentList).count -match '0|1') {
$SingleArgument = $True
}
Else {
$SingleArgument = $False
}
}
-match is a bug. it also matches 10 20 and so on there must be -le 1 !
may be this bug can affect on above error
btw, may be stuff like $ArgumentList -is [array] can be used instead ?
