[Feature Request] Parallel Dependency installations

[kilasuit]

When installing meta modules like Az / AzureRM each of the required modules gets installed sequentially which is painfully slow and could be greatly improved.

Therefore I would ask that as a feature request for PSGet v3 that dependencies could be installed via some form of parallelisation as to reduce overall installation time

[SteveL-MSFT]

It makes sense to install in parallel and resolve dependencies so they aren't installed multiple times (like in the case of Az modules all depending on Az.Accounts).

[JustinGrote]

My recommendation would be to basically just execute a NuGet restore, it already has all the logic in there to do this in parallel and handle all the dependencies. I'll reference my Modulefast POC again https://gist.github.com/JustinGrote/ecdf96b4179da43fb017dccbd1cc56f6 which installs all the Az modules (which are a really good test case due to their metapackages and dependencies) in less than 5 seconds

[o-l-a-v]

I did some experimentation with runspace pools yesterday. Pseudocode with the Az module:

• Find dependencies first: $Parent = Find-PSResource -Name 'Az'
• List of modules to install = $Parent.Name + $Parent.Dependencies
• Parallelize foreach with runspace pools, use Install-PSResource with -SkipDependencyCheck since we already have all dependencies.

This drastically sped up installation of a module with many dependecies, and should be backwards compatible to Windows PowerShell >= 3.

This is a low hanging fruit for great speed improvements IMO.

---

Edit: I created a function that uses runspace pools to install modules in parallel, and tested it on Az, Microsoft.Graph, Microsoft.Graph.Beta and all their dependencies, 166 unique modules in total (as of writing).

• Original Save-PSResource -Repository 'PSGallery' -TrustRepository -IncludeXml -SkipDependencyCheck -Path $SavePath -Name $ListOfModules took 110 seconds.
• Save-PSResourceInParallel -Repository 'PSGallery' -Path $SavePath -Name $ListOfModules -ThrottleLimit 16 using runpsace factory took around 20 seconds.
• Justin Grote ModuleFast Install-ModuleFast -ModulesToInstall 'Az','Microsoft.Graph','Microsoft.Graph.Beta' -Destination $SavePath -Credential ([PSCredential]::Empty) -NoPSModulePathUpdate -NoProfileUpdate -Update took about 35 seconds.

To my surprise, by parallalizing with runspace factory one can go significantly faster than even ModuleFast. And it's backwards compatible with Windows PowerShell.

Here is the code if anyone want to experiment further

# Function
function Save-PSResourceInParallel {
    <#
        .SYNOPSIS
            Speed up PSResourceGet\Save-PSResource by parallizing using PowerShell native runspace factory.

        .NOTES
            Author:   Olav Rønnestad Birkeland | github.com/o-l-a-v
            Created:  231116
            Modified: 231116

        .EXAMPLE
            Save-PSResourceInParallel -Type 'Module' -Name (Find-PSResource -Repository 'PSGallery' -Type 'Module' -Name 'Az').'Dependencies'.'Name'
    #>
    [CmdletBinding()]
    [OutputType([Microsoft.PowerShell.PSResourceGet.UtilClasses.PSResourceInfo[]])]
    Param(
        [Parameter(Mandatory)]
        [string[]] $Name,

        [Parameter()]
        [bool] $IncludeXml = [bool] $true,

        [Parameter()]
        [ValidateNotNullOrEmpty()]
        [string] $Path,

        [Parameter()]
        [ValidateNotNullOrEmpty()]
        [string] $PSResourceGetPath = (Get-Module -Name 'Microsoft.PowerShell.PSResourceGet').'Path',

        [Parameter()]
        [ValidateNotNullOrEmpty()]
        [string] $Repository = 'PSGallery',

        [Parameter()]
        [bool] $SkipDependencyCheck = [bool] $true,

        [Parameter()]
        [byte] $ThrottleLimit = 10,

        [Parameter()]
        [bool] $TrustRepository = [bool] $true
    )


    # Begin
    Begin {
        # Assets
        $ScriptBlock = [scriptblock]{
            [OutputType([System.Void])]
            Param(
                [Parameter()]
                [bool] $IncludeXml = $true,

                [Parameter(Mandatory)]
                [ValidateNotNullOrEmpty()]
                [string] $Name,

                [Parameter(Mandatory)]
                [ValidateNotNullOrEmpty()]
                [string] $Path,

                [Parameter(Mandatory)]
                [ValidateNotNullOrEmpty()]
                [string] $PSResourceGetPath,

                [Parameter(Mandatory)]
                [ValidateNotNullOrEmpty()]
                [string] $Repository,

                [Parameter()]
                [bool] $SkipDependencyCheck = $true,

                [Parameter()]
                [bool] $TrustRepository = $true
            )
            $ErrorActionPreference = 'Stop'
            $null = Import-Module -Name $PSResourceGetPath
            Microsoft.PowerShell.PSResourceGet\Save-PSResource -Repository $Repository -TrustRepository:$TrustRepository `
                -IncludeXml:$IncludeXml -Path $Path -SkipDependencyCheck:$SkipDependencyCheck -Name $Name
        }

        # Initilize runspace pool
        $RunspacePool = [runspacefactory]::CreateRunspacePool(1,$ThrottleLimit)
        $RunspacePool.Open()
    }


    # Process
    Process {
        # Start jobs in the runspace pool
        $RunspacePoolJobs = [PSCustomObject[]](
            $(
                foreach ($ModuleName in $Name) {
                    $PowerShellObject = [powershell]::Create().AddScript($ScriptBlock).AddParameters(
                        @{
                            'IncludeXml'          = [bool] $IncludeXml
                            'Name'                = [string] $ModuleName
                            'Path'                = [string] $Path
                            'PSResourceGetPath'   = [string] $PSResourceGetPath
                            'Repository'          = [string] $Repository
                            'SkipDependencyCheck' = [bool] $SkipDependencyCheck
                            'TrustRepository'     = [bool] $TrustRepository
                        }
                    )
                    $PowerShellObject.'RunspacePool' = $RunspacePool
                    [PSCustomObject]@{
                        'ModuleName' = $ModuleName
                        'Instance'   = $PowerShellObject
                        'Result'     = $PowerShellObject.BeginInvoke()
                    }
                }
            )
        )

        # Wait for jobs to finish
        $PrettyPrint = [string]('0'*$RunspacePoolJobs.'Count'.ToString().'Length')
        while ($RunspacePoolJobs.Where{-not $_.'Result'.'IsCompleted'}.'Count' -gt 0) {
            Write-Verbose -Message (
                '{0} / {1} jobs finished, {2} / {0} was successfull.' -f (
                    $RunspacePoolJobs.Where{$_.'Result'.'IsCompleted'}.'Count'.ToString($PrettyPrint),
                    $RunspacePoolJobs.'Count'.ToString(),
                    $RunspacePoolJobs.Where{$_.'Result'.'IsCompleted' -and -not $_.'Instance'.'HadErrors'}.'Count'.ToString($PrettyPrint)
                )
            )
            Start-Sleep -Milliseconds 250
        }

        # Get success state of jobs
        Write-Verbose -Message (
            $RunspacePoolJobs.ForEach{
                [PSCustomObject]@{
                    'Name'        = [string] $_.'ModuleName'
                    'IsCompleted' = [bool] $_.'Result'.'IsCompleted'
                    'HadErrors'   = [bool] $_.'Instance'.'HadErrors'
                }
            } | Sort-Object -Property 'ModuleName' | Format-Table | Out-String
        )

        # Collect results
        $Results = [Microsoft.PowerShell.PSResourceGet.UtilClasses.PSResourceInfo[]](
            $RunspacePoolJobs.ForEach{
                $_.'Instance'.EndInvoke($_.'Result')
            }
        )
    }


    # End
    End {
        # Terminate runspace pool
        $RunspacePool.Close()
        $RunspacePool.Dispose()

        # Output results
        $Results
    }
}



# Testing
if ($false) {
    # Import module
    Import-Module -Name 'Microsoft.PowerShell.PSResourceGet' -RequiredVersion '1.0.1'


    # Assets
    ## List of modules
    $ListOfModules = [string[]]('Az','Microsoft.Graph','Microsoft.Graph.Beta')
    $ListOfModules += [string[]]((Find-PSResource -Name $ListOfModules).'Dependencies'.'Name')
    $ListOfModules = [string[]]($ListOfModules | Sort-Object -Unique)

    ## Temp save path
    $SavePath  = [string][System.IO.Path]::Combine([System.Environment]::GetFolderPath('Desktop'),'Modules',[datetime]::Now.ToString('yyyyMMddHHmmss'))


    # Prepare temp save path
    if (-not [System.IO.Directory]::Exists($SavePath)) {
        $null = [System.IO.Directory]::CreateDirectory($SavePath)
    }


    # Install modules
    ## Original
    Measure-Command -Expression {
        Save-PSResource -Repository 'PSGallery' -TrustRepository -IncludeXml -SkipDependencyCheck -Path $SavePath -Name $ListOfModules
    }

    ## Parallel with Runspace Factory
    Measure-Command -Expression {
        Save-PSResourceInParallel -Repository 'PSGallery' -Path $SavePath -Name $ListOfModules -ThrottleLimit 16
    }

    ## ModuleFast
    ### Load if not already loaded
    & (
        [scriptblock]::Create(
            (
                Invoke-RestMethod -Method 'Get' -Uri 'https://raw.githubusercontent.com/JustinGrote/ModuleFast/main/ModuleFast.ps1'
            )
        )
    )
    ### Run
    Measure-Command -Expression {
        Install-ModuleFast -ModulesToInstall 'Az','Microsoft.Graph','Microsoft.Graph.Beta' -Destination $SavePath -Credential ([PSCredential]::Empty) -NoPSModulePathUpdate -NoProfileUpdate -Update
    }


    # Clean up
    ## For testing again
    [System.IO.Directory]::Delete($SavePath,$true);$null=[System.IO.Directory]::CreateDirectory($SavePath)

    ## For good
    [System.IO.Directory]::Delete($SavePath,$true)
}