PSResourceGet `Find-PSResource` with a string array of module names is slow: Bulk/batch the API calls?

Summary of the new feature / enhancement

Doing PowerShellGet\Find-PSResource -Type 'Module' -Repository 'PSGallery' -Name $InstalledModules is slow, seems like it queries one at a time.

The NuGet API can do multiple, like for instance (i know latest beta went away from using NuGet):

# Faster, multiple at a time
## Two manually
Invoke-RestMethod -Method 'Get' -Uri "https://www.powershellgallery.com/api/v2/Packages?`$filter=IsLatestVersion and IsPrerelease eq false and (Id eq 'Az.Accounts' or Id eq 'Microsoft.Graph.Authentication')&semVerLevel=1.0.0"
## Ten first from a string array of module names
Invoke-RestMethod -Method 'Get' -Uri (
    'https://www.powershellgallery.com/api/v2/Packages?$filter=IsLatestVersion and IsPrerelease eq false and (' +
            (
                $InstalledModules[0 .. 10].ForEach{
                    "Id eq '{0}'" -f $_
                } -join ' or '
            ) + ')&semVerLevel=1.0.0'
)

Here are some PowerShell to show how slow it is currently. I copied out a string array of module names.

Click to view

# String array of modules
$Modules = [string[]](
    'AIPService,AWSPowerShell.NetCore,Az,Az.Accounts,Az.ADDomainServices,Az.Advisor,Az.Aks,Az.AlertsManagement,Az.AnalysisServices,Az.ApiManagement,Az.App,Az.AppConfiguration,Az.ApplicationInsights,Az.ApplicationMonitor,Az.Attestation,Az.Automanage,Az.Automation,Az.BareMetal,Az.Batch,Az.Billing,Az.BillingBenefits,Az.Blueprint,Az.BootStrapper,Az.BotService,Az.Cdn,Az.ChangeAnalysis,Az.CloudService,Az.CognitiveServices,Az.Communication,Az.Compute,Az.Compute.ManagedService,Az.ConfidentialLedger,Az.Confluent,Az.ConnectedKubernetes,Az.ConnectedMachine,Az.ConnectedNetwork,Az.ContainerInstance,Az.ContainerRegistry,Az.CosmosDB,Az.CostManagement,Az.CustomLocation,Az.CustomProviders,Az.Dashboard,Az.DataBox,Az.DataBoxEdge,Az.Databricks,Az.Datadog,Az.DataFactory,Az.DataLakeAnalytics,Az.DataLakeStore,Az.DataMigration,Az.DataProtection,Az.DataShare,Az.DedicatedHsm,Az.DeploymentManager,Az.DesktopVirtualization,Az.DeviceProvisioningServices,Az.DeviceUpdate,Az.DevSpaces,Az.DevTestLabs,Az.DigitalTwins,Az.DiskPool,Az.Dns,Az.DnsResolver,Az.DynatraceObservability,Az.EdgeOrder,Az.Elastic,Az.ElasticSan,Az.EventGrid,Az.EventHub,Az.FluidRelay,Az.FrontDoor,Az.Functions,Az.GuestConfiguration,Az.HanaOnAzure,Az.HDInsight,Az.HealthBot,Az.HealthcareApis,Az.HPCCache,Az.ImageBuilder,Az.ImportExport,Az.Insights,Az.IotCentral,Az.IotHub,Az.KeyVault,Az.KubernetesConfiguration,Az.Kusto,Az.LabServices,Az.LoadTesting,Az.LogicApp,Az.Logz,Az.MachineLearning,Az.MachineLearningCompute,Az.MachineLearningServices,Az.Maintenance,Az.ManagedServiceIdentity,Az.ManagedServices,Az.ManagementPartner,Az.Maps,Az.MariaDb,Az.Marketplace,Az.MarketplaceOrdering,Az.Media,Az.Migrate,Az.MixedReality,Az.MobileNetwork,Az.Monitor,Az.MonitoringSolutions,Az.MySql,Az.NetAppFiles,Az.Network,Az.NetworkFunction,Az.Nginx,Az.NotificationHubs,Az.OperationalInsights,Az.Orbital,Az.Peering,Az.PolicyInsights,Az.Portal,Az.PostgreSql,Az.PowerBIEmbedded,Az.PrivateDns,Az.Profile,Az.ProviderHub,Az.Purview,Az.Quota,Az.RecoveryServices,Az.RedisCache,Az.RedisEnterpriseCache,Az.Relay,Az.Reservations,Az.ResourceGraph,Az.ResourceMover,Az.Resources,Az.Search,Az.Security,Az.SecurityInsights,Az.ServiceBus,Az.ServiceFabric,Az.ServiceLinker,Az.SignalR,Az.SpringCloud,Az.Sql,Az.SqlVirtualMachine,Az.Ssh,Az.StackEdge,Az.StackHCI,Az.StackHCI.NetworkHUD,Az.Storage,Az.StorageMover,Az.StorageSync,Az.StreamAnalytics,Az.Subscription,Az.Support,Az.Synapse,Az.Tags,Az.TimeSeriesInsights,Az.Tools.Installer,Az.Tools.Migration,Az.Tools.Predictor,Az.TrafficManager,Az.VMware,Az.VoiceServices,Az.Websites,Az.WindowsIotServices,AzSK,AzSK.AAD,AzSK.ADO,AzSK.AzureDevOps,Azure,Azure.AnalysisServices,Azure.Storage,AzureAD,AzureADPreview,AzureRM.profile,AzViz,ConfluencePS,DefenderMAPS,Evergreen,ExchangeOnlineManagement,GetBIOS,ImportExcel,Intune.USB.Creator,IntuneBackupAndRestore,Invokeall,JWTDetails,Mailozaurr,Microsoft.Graph,Microsoft.Graph.Applications,Microsoft.Graph.Authentication,Microsoft.Graph.Bookings,Microsoft.Graph.Calendar,Microsoft.Graph.ChangeNotifications,Microsoft.Graph.CloudCommunications,Microsoft.Graph.Compliance,Microsoft.Graph.CrossDeviceExperiences,Microsoft.Graph.DeviceManagement,Microsoft.Graph.DeviceManagement.Actions,Microsoft.Graph.DeviceManagement.Administration,Microsoft.Graph.DeviceManagement.Enrolment,Microsoft.Graph.DeviceManagement.Functions,Microsoft.Graph.Devices.CloudPrint,Microsoft.Graph.Devices.CorporateManagement,Microsoft.Graph.Devices.ServiceAnnouncement,Microsoft.Graph.DirectoryObjects,Microsoft.Graph.Education,Microsoft.Graph.Files,Microsoft.Graph.Financials,Microsoft.Graph.Groups,Microsoft.Graph.Identity.DirectoryManagement,Microsoft.Graph.Identity.Governance,Microsoft.Graph.Identity.SignIns,Microsoft.Graph.Intune,Microsoft.Graph.Mail,Microsoft.Graph.ManagedTenants,Microsoft.Graph.Notes,Microsoft.Graph.People,Microsoft.Graph.PersonalContacts,Microsoft.Graph.Planner,Microsoft.Graph.Reports,Microsoft.Graph.SchemaExtensions,Microsoft.Graph.Search,Microsoft.Graph.Security,Microsoft.Graph.Sites,Microsoft.Graph.Teams,Microsoft.Graph.Users,Microsoft.Graph.Users.Actions,Microsoft.Graph.Users.Functions,Microsoft.Graph.WindowsUpdates,Microsoft.Online.SharePoint.PowerShell,Microsoft.PowerShell.ConsoleGuiTools,Microsoft.PowerShell.SecretManagement,Microsoft.PowerShell.SecretStore,Microsoft.RDInfra.RDPowershell,Microsoft.RdInfra.RDPowershell.Migration,MicrosoftGraphSecurity,MicrosoftPowerBIMgmt,MicrosoftPowerBIMgmt.Admin,MicrosoftPowerBIMgmt.Capacities,MicrosoftPowerBIMgmt.Data,MicrosoftPowerBIMgmt.Profile,MicrosoftPowerBIMgmt.Reports,MicrosoftPowerBIMgmt.Workspaces,MicrosoftTeams,MSAL.PS,MSGraphFunctions,MSOnline,Nevergreen,newtonsoft.json,Office365DnsChecker,Optimized.Mga,Optimized.Mga.AzureAD,Optimized.Mga.Mail,Optimized.Mga.Report,Optimized.Mga.SharePoint,PackageManagement,PartnerCenter,PartnerCenter.NetCore,Pester,platyPS,PnP.PowerShell,PolicyFileEditor,PoshRSJob,PowerShellGet,PSGraph,PSIntuneAuth,PSPackageProject,PSPKI,PSReadLine,PSScriptAnalyzer,PSWindowsUpdate,RunAsUser,SetBIOS,SharePointPnPPowerShellOnline,SHiPS,SpeculationControl,Trackyon.Utils,VSTeam,WindowsAutoPilotIntune'.Split(',')
)


# Microsoft.PowerShell.PSResourceGet
## Get
### Using Cmdlet as is
Measure-Command -Expression {
    $Script:PSResourceGetResults = [array](
        Microsoft.PowerShell.PSResourceGet\Find-PSResource -Type 'Module' -Repository 'PSGallery' -Name $Modules | Sort-Object -Property 'Name' -Unique
    )
}
### Using `ForEach-Object -Parallel`
Measure-Command -Expression {
    $Script:PSResourceGetResults = [array](
        $Modules | ForEach-Object -Parallel {
            Microsoft.PowerShell.PSResourceGet\Find-PSResource -Type 'Module' -Repository 'PSGallery' -Name $_
        } -ThrottleLimit 50 | Sort-Object -Property 'Name' -Unique
    )
}

## Present results
$PSResourceGetResults.ForEach{
    [PSCustomObject]@{
        'Name' = [string] $_.'Name'
        'Author' = [string] $_.'Author'
        'Version' = [System.Version] $_.'Version'
        'NormalizedVersion' = [System.Version] $_.'AdditionalMetadata'.'NormalizedVersion'
    }
} | Format-Table -AutoSize



# Bulk PowerShellGallery NuGet API
## Get
Measure-Command -Expression {
    $PageSize = [byte] 30
    $Page = [byte] 1
    $Script:PowerShellGalleryNuGetAPIResults = [PSCustomObject[]](
        $(
            do {
                $FromIndex = [uint16](($Page-1) * $PageSize)
                $ToIndex   = [uint16]($Page * $PageSize -lt $Modules.'Count' ? $Page * $PageSize : $Modules.'Count')
                Invoke-RestMethod -Method 'Get' -Uri (
                    'https://www.powershellgallery.com/api/v2/Packages?$filter=IsLatestVersion and IsPrerelease eq false and (' +
                        (
                            $Modules[$FromIndex .. $ToIndex].ForEach{
                                "Id eq '{0}'" -f $_
                            } -join ' or '
                        ) + ')&semVerLevel=1.0.0'
                )
                $Page++
            }
            until ($ToIndex -ge $InstalledModules.'Count')
        ) | Sort-Object -Property @{'Expression'={$_.'title'.'#text'}} -Unique
    )
}

## Present results
$PowerShellGalleryNuGetAPIResults.ForEach{
    [PSCustomObject]@{
        'Name' = [string] $_.'title'.'#text'
        'Author' = [string] $_.'author'.'name'
        'Version' = [System.Version] $_.'properties'.'Version'
        'NormalizedVersion' = [System.Version] $_.'properties'.'NormalizedVersion'
    }
} | Format-Table -AutoSize

Proposed technical implementation details (optional)

Use API to call multiple modules at a time?
Multithreading? / Jobs?

Apr 02 '23 10:04 o-l-a-v

Thanks @o-l-a-v we have improvements planned

Apr 03 '23 18:04 SydneyhSmith

Did some experimenting with runspace pools for Find-PSResource on Az, Microsoft.Graph, Microsoft.Graph.Beta and all their dependencies, 166 unique modules in total.

Find-PSResource -Repository 'PSGallery' -Type 'Module' -Name $ListOfModules took 26 seconds.
Find-PSResourceInParallel -Repository 'PSGallery' -Type 'Module' -Name $ListOfModules -ThrottleLimit 25 took 1.6 seconds.
Batching API request with 30 module names in one request: 1.9 seconds.
Combined: Haven't tested.

We're talking big savings if both parallelizing and batching API requests. And both will work with Windows PowerShell.

Code if others want to experiment

# Function
function Find-PSResourceInParallel {
    <#
        .SYNOPSIS
            Speed up PSResourceGet\Find-PSResource by parallizing using PowerShell native runspace factory.

        .NOTES
            Author:   Olav Rønnestad Birkeland | github.com/o-l-a-v
            Created:  231116
            Modified: 231116

        .EXAMPLE
            Find-PSResourceInParallel -Type 'Module' -Name (Find-PSResource -Repository 'PSGallery' -Type 'Module' -Name 'Az').'Dependencies'.'Name'
    #>
    [CmdletBinding()]
    [OutputType([Microsoft.PowerShell.PSResourceGet.UtilClasses.PSResourceInfo[]])]
    Param(
        [Parameter(Mandatory)]
        [string[]] $Name,

        [Parameter()]
        [ValidateNotNullOrEmpty()]
        [string] $PSResourceGetPath = (Get-Module -Name 'Microsoft.PowerShell.PSResourceGet').'Path',

        [Parameter()]
        [ValidateNotNullOrEmpty()]
        [string] $Repository = 'PSGallery',

        [Parameter()]
        [byte] $ThrottleLimit = 10,

        [Parameter()]
        [ValidateNotNullOrEmpty()]
        [ValidateSet('Module','Script')]
        [string] $Type = 'Module'
    )


    # Begin
    Begin {
        # Assets
        $ScriptBlock = [scriptblock]{
            [OutputType([Microsoft.PowerShell.PSResourceGet.UtilClasses.PSResourceInfo])]
            Param(
                [Parameter(Mandatory)][ValidateNotNullOrEmpty()][string]$Name,
                [Parameter(Mandatory)][ValidateNotNullOrEmpty()][string]$PSResourceGetPath,
                [Parameter(Mandatory)][ValidateNotNullOrEmpty()][string]$Repository,
                [Parameter(Mandatory)][ValidateNotNullOrEmpty()][string]$Type
            )
            $ErrorActionPreference = 'Stop'
            $null = Import-Module -Name $PSResourceGetPath
            Microsoft.PowerShell.PSResourceGet\Find-PSResource -Repository $Repository -Type $Type -Name $Name
        }

        # Initilize runspace pool
        $RunspacePool = [runspacefactory]::CreateRunspacePool(1,$ThrottleLimit)
        $RunspacePool.Open()
    }


    # Process
    Process {
        # Start jobs in the runspace pool
        $RunspacePoolJobs = [PSCustomObject[]](
            $(
                foreach ($ModuleName in $Name) {
                    $PowerShellObject = [powershell]::Create().AddScript($ScriptBlock).AddParameters(
                        @{
                            'PSResourceGetPath' = [string] $PSResourceGetPath
                            'Repository' = [string] $Repository
                            'Type'       = [string] $Type
                            'Name'       = [string] $ModuleName
                        }
                    )
                    $PowerShellObject.'RunspacePool' = $RunspacePool
                    [PSCustomObject]@{
                        'ModuleName' = $ModuleName
                        'Instance'   = $PowerShellObject
                        'Result'     = $PowerShellObject.BeginInvoke()
                    }
                }
            )
        )

        # Wait for jobs to finish
        $PrettyPrint = [string]('0'*$RunspacePoolJobs.'Count'.ToString().'Length')
        while ($RunspacePoolJobs.Where{-not $_.'Result'.'IsCompleted'}.'Count' -gt 0) {
            Write-Verbose -Message (
                '{0} / {1} jobs finished, {2} / {0} was successfull.' -f (
                    $RunspacePoolJobs.Where{$_.'Result'.'IsCompleted'}.'Count'.ToString($PrettyPrint),
                    $RunspacePoolJobs.'Count'.ToString(),
                    $RunspacePoolJobs.Where{$_.'Result'.'IsCompleted' -and -not $_.'Instance'.'HadErrors'}.'Count'.ToString($PrettyPrint)
                )
            )
            Start-Sleep -Milliseconds 250
        }

        # Get success state of jobs
        Write-Verbose -Message (
            $RunspacePoolJobs.ForEach{
                [PSCustomObject]@{
                    'Name'        = [string] $_.'ModuleName'
                    'IsCompleted' = [bool] $_.'Result'.'IsCompleted'
                    'HadErrors'   = [bool] $_.'Instance'.'HadErrors'
                }
            } | Sort-Object -Property 'ModuleName' | Format-Table | Out-String
        )

        # Collect results
        $Results = [Microsoft.PowerShell.PSResourceGet.UtilClasses.PSResourceInfo[]](
            $RunspacePoolJobs.ForEach{
                $_.'Instance'.EndInvoke($_.'Result')
            }
        )
    }


    # End
    End {
        # Terminate runspace pool
        $RunspacePool.Close()
        $RunspacePool.Dispose()

        # Output results
        $Results
    }
}



# Testing
if ($false) {
    # Import PSResourceGet
    Import-Module -Name 'Microsoft.PowerShell.PSResourceGet' -RequiredVersion '1.0.1'


    # Assets
    $ListOfModules = [string[]]('Az','Microsoft.Graph','Microsoft.Graph.Beta')
    $ListOfModules += [string[]]((Find-PSResource -Name $ListOfModules).'Dependencies'.'Name')
    $ListOfModules = [string[]]($ListOfModules | Sort-Object -Unique)


    # Test
    Find-PSResourceInParallel -Name $ListOfModules -ThrottleLimit 20
    Find-PSResourceInParallel -Name 'Az.*'


    # Measure / Compare
    ## Original
    Measure-Command -Expression {
        Find-PSResource -Repository 'PSGallery' -Type 'Module' -Name $ListOfModules
    }

    ## Parallel with Runspace Factory
    Measure-Command -Expression {
        Find-PSResourceInParallel -Repository 'PSGallery' -Type 'Module' -Name $ListOfModules -ThrottleLimit 25
    }

    ## Batch API manually sequentially
    ### Measure
    Measure-Command -Expression {
        $PageSize = [byte] 30
        $Page = [byte] 1
        $Script:PowerShellGalleryNuGetAPIResults = [PSCustomObject[]](
            $(
                do {
                    $FromIndex = [uint16](($Page-1) * $PageSize)
                    $ToIndex   = [uint16]($Page * $PageSize -lt $ListOfModules.'Count' ? $Page * $PageSize : $ListOfModules.'Count')
                    Invoke-RestMethod -Method 'Get' -Uri (
                        'https://www.powershellgallery.com/api/v2/Packages?$filter=IsLatestVersion and IsPrerelease eq false and (' +
                            (
                                $ListOfModules[$FromIndex .. $ToIndex].ForEach{
                                    "Id eq '{0}'" -f $_
                                } -join ' or '
                            ) + ')&semVerLevel=1.0.0'
                    )
                    $Page++
                }
                until ($ToIndex -ge $ListOfModules.'Count')
            ) | Sort-Object -Property @{'Expression'={$_.'title'.'#text'}} -Unique
        )
    }
    ### Present results
    $PowerShellGalleryNuGetAPIResults.ForEach{
        [PSCustomObject]@{
            'Name' = [string] $_.'title'.'#text'
            'Author' = [string] $_.'author'.'name'
            'Version' = [System.Version] $_.'properties'.'Version'
            'NormalizedVersion' = [System.Version] $_.'properties'.'NormalizedVersion'
        }
    } | Format-Table -AutoSize
}

Nov 16 '23 19:11 o-l-a-v

PSResourceGet PSResourceGet copied to clipboard

`Find-PSResource` with a string array of module names is slow: Bulk/batch the API calls?

Summary of the new feature / enhancement

Proposed technical implementation details (optional)

PSResourceGet
PSResourceGet copied to clipboard