Get-PnPRecycleBinItem exceeds the list view threshold

[DanielSanIT]

Reporting an Issue or Missing Feature

"Get-PnPRecycleBinItem" command on the recyclebin with a lot of items fails with "the list view threshold" error.

I have tried to filter results by following commands such as "Get-PnPRecycleBinItem | select -last 10" and "Get-PnPRecycleBinItems | ? DeletedDate -gt $today" however, these commands did not help. I got the same error.

This worked for recycle bin with more than 40k items some time ago.

Is there any workaround solution? Is it possible to fix this trouble in PnP-PowerShell module?

Expected behavior

Restored files/files list

Actual behavior

Error message: "Get-PnPRecycleBinItem : The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator. At line:1 char:2

• (Get-PnPRecycleBinItem).count

•  + CategoryInfo          : WriteError: (:) [Get-PnPRecycleBinItem], ServerException
   + FullyQualifiedErrorId : EXCEPTION,SharePointPnP.PowerShell.Commands.RecycleBin.GetRecycleBinItems"
  
  

Steps to reproduce behavior

Connect-PnPOnline -Url https://... (Get-PnPRecycleBinItem).count

Which version of the PnP-PowerShell Cmdlets are you using?

• [ ] PnP PowerShell for SharePoint 2013
• [ ] PnP PowerShell for SharePoint 2016
• [ x] PnP PowerShell for SharePoint Online

What is the version of the Cmdlet module you are running?

3.1.1809.0

How did you install the PnP-PowerShell Cmdlets?

• [ ] MSI Installed downloaded from GitHub
• [ x] Installed through the PowerShell Gallery with Install-Module
• [ ] Other means

[ghost]

Thank you for reporting this issue. We will be triaging your incoming issue as soon as possible.

[CollinChaffin]

Is this being looked at yet? It's been a month and a half and this module is simply 100% broken and can no longer even so much as get a COUNT of recycle bin items etc.

[KrystianNiepsuj]

Hello @CollinChaffin , as part of the January release I pushed an update to this cmdlet that should resolve the issue you were facing. I, too, was facing this problem on my own tenant. Unfortunately, if the recycle bin is too full, the service begins to misbehave a bit, so there is no simple fix to retrieve all items.

That said, i added a parameter to allow you to limit the amount of returned results, which you should be able to use to retrieve items without the service timing out for you. Just use the -RowLimit parameter for this function. I recommend 5000 to 10000 items to return, that seems to be the sweet spot - depending on how the tenant is doing you may be able to retrieve more (but, again, performance will vary tenant by tenant).

Hope it helps!

[DanielSanIT]

Hi @MrDoNotBreak , Thanks for your fix.

Two questions for you.

Do you know when we will get the update to see this fix?

As I understand, we just will get top 5000-10000 results. Is there any pagination, to get all result by 5-10k elements on page?

[6079-Winston-Smith]

@MrDoNotBreak this problem also seems to apply to Restore-PnPRecycleBinItem, for which there is no limiting parameter. For example, if I try:

Get-PnPRecycleBinItem -SecondStage -RowLimit 4500 | ? {($_.DeletedByEmail -eq '[email protected]')} | Restore-PnPRecycledBinItem -Force

Then errors are thrown: Restore-PnPRecycleBinItem : The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator.

[Brendenawit]

Hello @CollinChaffin , as part of the January release I pushed an update to this cmdlet that should resolve the issue you were facing. I, too, was facing this problem on my own tenant. Unfortunately, if the recycle bin is too full, the service begins to misbehave a bit, so there is no simple fix to retrieve all items.

@MrDoNotBreak thank you for putting the restriction on the get-pnprecyclebinitem. Unfortunately I am having the same issue as @6079-Winston-Smith where a client has accidentally deleted a massive amount of files and the restore function fails because of the 5000 limit, regardless of how few files are piped to it. Even if I have to restore items 5000 at a time it would be a big help to have -RowLimit on the restore function. The only other alternative is to do a full library restore and lose the day's work.

[pmatthews05]

@Brendenawit I have come across the same limitation recently. It seems to be that the Restore-PnpRecycleBinItem does a call to get-pnpRecycleBinItem in the code, but this code doesn't have a Row Limit on it. Until someone at PNP fixes this issue, (not meaning to promote myself, but it should help) I wrote a blog on this yesterday., with code to restore.

[Professr]

Clear-PnPRecycleBinItem has the same problem, so I've added support for the RowLimit parameter to it. Would this be useful to make a PR for?

[KoenZomers]

All recycle bin cmdlets should have the RowLimit parameter in the July 2020 release.

[PrzemyslawKlys]

And when can we expect that release? I am just trying to restore files and hitting this error.

[KoenZomers]

@PrzemyslawKlys, here's a custom build which includes this fix already, if you want to test it out already:

SharePointPnPPowerShellOnline.zip

Official July 2020 release will be somewhere between next Friday and the end of next week. Exact date yet to be defined.

[KoenZomers]

July 13th will be the release date

[PrzemyslawKlys]

Thank you!

[Vindicator007]

Perhaps if I have to ask this question I shouldn't be testing your custom build, but if I've installed the module from the gallery, how do I incorporate the changes from your zip file above? I've tried copying the folder over to C:\Program Files\WindowsPowerShell\Modules but I still don't see the -RowLimit option on Restore-PnPRecycleBinItem. TIA

[KoenZomers]

Questions are always welcome @Vindicator007 . As I think more people will have the same question, I have just written a wiki page outlining how you can make use of this or other custom builds which we share in the issue topics:

https://github.com/pnp/PnP-PowerShell/wiki/Using-custom-PnP-PowerShell-builds

[Vindicator007]

@KoenZomers Thanks for the wiki, I was off by one directory, but after uninstall and following your post it still doesn't seem to be working for me. It acts like an untrusted script whenever I attempt to use tab completion, prompting me to allow it to run even though I tried setting execution policy to unrestricted. Maybe it's me, but I thought I would let you know.

[PrzemyslawKlys]

@Vindicator007 You may need to right-click on the file in explorer, properties, and check if you need to unblock them. It's often like that when downloaded from internet directly.

[PrzemyslawKlys]

@KoenZomers any reason why pipeline doesn't work

$Stage = Get-PnPRecycleBinItem -SecondStage -RowLimit 150000
$Files = $Stage | Where-Object { $_.DirName -like "$Test*"} #| Out-HtmlView -ScrollX -DisablePaging

#foreach ($File in $Files) {
#    $File | Restore-PnPRecycleBinItem -RowLimit 150000
#}

$Files | Restore-PnPRecycleBinItem -RowLimit 150000

Restore-PnPRecycleBinItem : The input object cannot be bound to any parameters for the command either because the command does not take pipeline input or the input and its properties do not match any of the parameters that take pipeline input.
At line:1 char:10
+ $Files | Restore-PnPRecycleBinItem -RowLimit 150000
+          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidArgument: (Microsoft.Share....RecycleBinItem:PSObject) [Restore-PnPRecycleBinItem], ParameterBindingException
    + FullyQualifiedErrorId : InputObjectNotBound,SharePointPnP.PowerShell.Commands.RecycleBin.RestoreRecycleBinItem

It doesn't say it accepts pipeline like in other cmdlets, but examples show it's normally used that way.

get-help Restore-PnPRecycleBinItem -Full


NAME
    Restore-PnPRecycleBinItem

SYNOPSIS
    Restores the provided recycle bin item to its original location


SYNTAX
    Restore-PnPRecycleBinItem [-Force <SwitchParameter>] [-RowLimit <Int32>] [-Connection <PnPConnection>] [<CommonParameters>]

    Restore-PnPRecycleBinItem -Identity <RecycleBinItemPipeBind> [-Force <SwitchParameter>] [-Connection <PnPConnection>] [<CommonParameters>]


DESCRIPTION


PARAMETERS
    -Force [<SwitchParameter>]
        If provided, no confirmation will be asked to restore the recycle bin item

        Required?                    false
        Position?                    named
        Default value
        Accept pipeline input?
        Accept wildcard characters?

    -Identity <RecycleBinItemPipeBind>
        Id of the recycle bin item or the recycle bin item object itself to restore

        Required?                    true
        Position?                    named
        Default value
        Accept pipeline input?
        Accept wildcard characters?

    -RowLimit [<Int32>]
        Limits restoration to specified number of items

        Required?                    false
        Position?                    named
        Default value
        Accept pipeline input?
        Accept wildcard characters?

    -Connection [<PnPConnection>]
        Optional connection to be used by the cmdlet. Retrieve the value for this parameter by either specifying -ReturnConnection on Connect-PnPOnline or by executing Get-PnPConnection.

        Required?                    false
        Position?                    named
        Default value
        Accept pipeline input?
        Accept wildcard characters?

    <CommonParameters>
        This cmdlet supports the common parameters: Verbose, Debug,
        ErrorAction, ErrorVariable, WarningAction, WarningVariable,
        OutBuffer, PipelineVariable, and OutVariable. For more information, see
        about_CommonParameters (https:/go.microsoft.com/fwlink/?LinkID=113216).

INPUTS

OUTPUTS




    ------------------EXAMPLE 1---------------------

    PS:> Restore-PnpRecycleBinItem -Identity 72e4d749-d750-4989-b727-523d6726e442

    Restores the recycle bin item with Id 72e4d749-d750-4989-b727-523d6726e442 to its original location

    ------------------EXAMPLE 2---------------------

    PS:> Get-PnPRecycleBinItem | ? -Property LeafName -like "*.docx" | Restore-PnpRecycleBinItem

    Restores all the items in the first and second stage recycle bins to their original location of which the filename ends with the .docx extension

    ------------------EXAMPLE 3---------------------

    PS:> Restore-PnPRecycleBinItem -All -RowLimit 10000

    Permanently restores up to 10,000 items in the recycle bin


RELATED LINKS
    SharePoint Developer Patterns and Practices: http://aka.ms/sppnp

Did something change?

[PrzemyslawKlys]

Ok, so the reason for the above is, RowLimit was added, but only if you don't use Identity. This makes no sense :-(

[Vindicator007]

@PrzemyslawKlys Unblock did the trick, thanks. Now I'm getting the same thing you are, that it can't take pipeline input.

[KoenZomers]

Good feedback guys. I've updated the wiki with the instruction to unblock the files. Regarding it not working as expected, I guess that shows the value of sharing custom builds before it gets released into dev :)

I'm going to write a simple script to upload 5050 items to a doclib and delete them so I can try to reproduce the scenario.

[PrzemyslawKlys]

I am now trying to restore items between 22939-24209. I am now scrolling in web interface every 200 objects. I am slowly dying inside :-)

I can tell now that you simply need to update parameter sets. You have 2 parameter sets, where Identity is in a different parameter sets then RowLimit. It should be as simple as fix that there is a single parameter set name, not two.

Restore-PnPRecycleBinItem [-Force <SwitchParameter>] [-RowLimit <Int32>] [-Connection <PnPConnection>] [<CommonParameters>]

Restore-PnPRecycleBinItem -Identity <RecycleBinItemPipeBind> [-Force <SwitchParameter>] [-Connection <PnPConnection>] [<CommonParameters>]

[Vindicator007]

@KoenZomers Thanks for your efforts. Are there any updates forthcoming, before I click the 1 week restore button in the portal? 🤦‍♂️

[KoenZomers]

Haven't made progess on this one yet. I let a script run to upload 5050 files and then deleted them, but forgot to include the -Recycle argument in Remove-PnPListItem, so they bypassed the recycle bin. Going to upload the 5050 again now :)

[KoenZomers]

I can't seem to reproduce the issue here. I uploaded 5050 files, deleted them to the first stage recycle bin and then ran:

Restore-PnPRecycleBinItem

This worked fine and restored all 5050 items again.

Is the problem specifically to piping recyclebin items into Restore-PnPRecycleBinItem? In that case it would take -Identity as the parameter and that indeed does not allow a -RowLimit as it would only retrieve that single item anyway. Also that seems to be working fine here.

What am I missing?

[Brendenawit]

I apologise to anyone who's here looking for my previous comment that I deleted because I realised it was not helpful. The problem from the start has been that even when you pipe a single file to the restore function, the function would retrieve the entire list anyway (running into the 5000+ limit error). So the version of the function that accepts pipe input needs to limit its own retrieval function, or be rewritten in a way that does not retrieve the entire list before filtering it down to the ones you want to delete. The problem then is, if the retrieval function within restore-pnprecyclebinitem limits itself to 5000, what if the file it needs to delete is #5001?

So the test is (was):

1. Have 5050 items in the recycle bin
2. Produce a list of several of them
3. Pipe that list to Restore-PnPRecycleBinItem

In the broken scenario at the time of my original reply, this fails with {exceeds the list view threshold} because step 3 loaded the entire list from step 1 into a view, regardless as to what you piped to it. If the problem has been resolved, then @PrzemyslawKlys should be able to pipe a single file at a time to Restore-PnPRecycleBinItem without the error occurring as it did before, OR -RowLimit should be applicable to the version that accepts piped input so that limit can be passed through to the internal get-pnpRecycleBinItem.

[Brendenawit]

OR -RowLimit should be applicable to the version that accepts piped input so that limit can be passed through to the internal get-pnpRecycleBinItem.

But of course, then you run into the issue of "What if the file I'm looking for is #5001" but that's for smarter minds to figure out.

[PrzemyslawKlys]

To me, Restore/Remove commands needs work because:

• it should support a single item to be restored without -RowLimit. As far as I understand RowLimit it would mean in the background you're actually trying to get all 150000 items (when I request it) and then restore a single item.
• it should support a single or multiple items restores based on the ID of the item - I already have all the items I need since i used Get, we shouldn't need to request again.

If you go into the web to restore something, you click 5 checkboxes and press restore and it restores them in 5 seconds. or less.

In my tenant, I've over 150k items deleted (I've not checked further) and I needed to restore 70 of them based on a path.

# Get all items from recycle bin - although it would be cool if there would be -Filter option to not store 150k items in variable as AD cmdlets have
$Stage = Get-PnPRecycleBinItem -SecondStage -RowLimit 150000
# Here I'm limiting the outputt and test it with PSWriteHTML function Out-HtmlView to see if what I want to restore is there
$Files = $Stage | Where-Object { $_.DirName -like "$Test*"} #| Out-HtmlView -ScrollX -DisablePaging

# Here I simply go item by item and restore it, i shouldn't have to use pipeline or rowlimit
foreach ($File in $Files) {
    $File | Restore-PnPRecycleBinItem
}

# Here's of course pipeline version where $Files could have 1 item, 5 items 50k items and it should restore them (without rowlimit)
$Files | Restore-PnPRecycleBinItem

As you see above the RowLimit is a bit special - why would it be needed for Restore/Remove commands if you already have the data using Get command? You should simply pass whole item as pipeline or as parameter or alternatively accept ID as pipeline/parameter - without executing in background another get of 150k (or more elements - just to get 5 items).

[PrzemyslawKlys]

I can't seem to reproduce the issue here. I uploaded 5050 files, deleted them to the first stage recycle bin and then ran:

Restore-PnPRecycleBinItem

This worked fine and restored all 5050 items again.

Is the problem specifically to piping recyclebin items into Restore-PnPRecycleBinItem? In that case it would take -Identity as the parameter and that indeed does not allow a -RowLimit as it would only retrieve that single item anyway. Also that seems to be working fine here.

What am I missing?

You're doing it wrong ;-) Using Restore-PnPRecycleBinItem without any parameters or Restore-PnPRecycleBinItem -RowLimit 15000 will restore all 15000 items. We're trying to restore 1, 5, 1000 items that we want. So you need a way to pass Items/IDs to Restore and not even need RowLimit for that, as you already have them on you, and you know what you want to delete.

On the web interface when you try to restore 5 items that are on 7 different pages it's a nightmare. I had to switch to classic view because modern view refreshes automatically loosing track. On classic I was able to go into proper place but very time I pressed restore on an item it went to the beginning of the list. And then I had to click 200+ (wait 5 seconds to change page) times to get to items on a list 22000 choose 5 items and restore again, and repeat.

[KoenZomers]

@PrzemyslawKlys that sounds exactly like what I said:

Is the problem specifically to piping recyclebin items into Restore-PnPRecycleBinItem? In that case it would take -Identity as the parameter and that indeed does not allow a -RowLimit as it would only retrieve that single item anyway. Also that seems to be working fine here.

Can you share an exact PnP PowerShell line which returns an error for you?