[BUG] `Get-PnPListItem` causes `out of memory` exceptions even with `-ScriptBlock` and `-PageSize`

[Ruslan-Urban]

The Get-PnPListItem commandlet with -ScriptBlock does not need to write the list of items into the output because this is causing Out of memory exceptions. The intent of the combination of the -PageSize and the -ScriptBlock parameters is to process smaller batches to minimize the memory footprint. However, WriteOutput causes unintended object accumulation in the output stream, which is not being used in the vast majority of cases. This makes it impossible to scan the contents of huge document libraries containing hundreds of thousands of documents (e.g., to create inventory reports, permission reports, etc.). This is especially important for Azure Automation jobs that have limited resources.

Release: 3.1.0 Code version: 50fc8b3 File: src/Commands/Lists/GetListItem.cs

Suggested edits:

Lines 184 -189

                    WriteObject(listItems, true);

                    if (ScriptBlock != null)
                    {
                        ScriptBlock.Invoke(listItems);
                    }

Change to

                    if (ScriptBlock != null)
                    {
                        ScriptBlock.Invoke(listItems);
                    }
                    else
                    {
                        WriteObject(listItems, true);
                    }

[jackpoz]

I'm curious, does | Out-Null in PowerShell affect the memory usage at all ?

[Ruslan-Urban]

After thinking about it, I realized that the script block can also be used for filtering. Thus, the following code would avoid out of memory exceptions if only a small subset of rows can be returned.

                    if (ScriptBlock != null)
                    {
                        WriteObject(ScriptBlock.Invoke(listItems), true);
                    }
                    else
                    {
                        WriteObject(listItems, true);
                    }

Usage example:

$dispositionReviewCandidates = Get-PnPListItem -List "Documents" -Fields ("PublishTime") -PageSize 500 -ScriptBlock {
  param($listItems)
  $expiryDate = (GetDate).AddYears(-7);
  return ($listItems | ? {$_.FieldValues.PublishTime -le $expiryDate});
}