[BUG] Get-PnPFile throws "Does Not Exist" error when source path contains "+" (plus symbol)

[JazBInKC]

Reporting an Issue or Missing Feature

When using Get-PnPFile to write a file from SharePoint to NTFS the cmdlet throws a <UNC> does not exist error when any portion of the path or filename contains a "+"

Early versions of SharePoint PnP didn't have this issue as I seeded the data I'm trying to update with the exact code that's failing now.

Expected behavior

I expect Get-PnPFile to treat source and destination paths as -LiteralPath (or include -LiteralPath in the cmdlet).

Actual behavior

In the image below my Write-Host output shows "+" in the file path. The Get-PnPFile error shows spaces where the "+" should be.

Steps to reproduce behavior

Code generating the above screenshot:

$serverrelativeurl = $item.FieldValues.FileRef

       `$destinationfolder = $prefix + $item.FieldValues.FileDirRef.Replace("/","\")
        $serverrelativeurl = $item.FieldValues.FileRef 
        write-host "Writing:" $targetpath -ForegroundColor Green
        $result = Get-PnPFile -ServerRelativeUrl $serverrelativeurl -Path $destinationfolder -FileName $filename -AsFile -Force
        if (Test-Path -LiteralPath $targetpath) {
            (Get-Item -LiteralPath $targetpath -ErrorAction Continue).LastWriteTime = $modified
            $acl = Get-Acl -LiteralPath $targetpath
            $acl.SetOwner([System.Security.Principal.NTAccount] "Domain Admins")
            Set-Acl -LiteralPath $targetpath -AclObject $acl
            $global:writes += 1
        }
        else {
            write-host "ERROR Writing:" $targetpath -ForegroundColor Red`

What is the version of the Cmdlet module you are running?

PnP.PowerShell 1.10.24 PS5.1 Server 2016

Which operating system/environment are you running PnP PowerShell on?

• [ x] Windows
• [ ] Linux
• [ ] MacOS
• [ ] Azure Cloud Shell
• [ ] Azure Functions
• [ ] Other : please specify

[veronicageek]

@JazBInKC

PnP.PowerShell 1.10.24 PS5.1 Server 2016 Early versions of SharePoint PnP didn't have this issue

As you seem to use SharePoint on-prem (that we don't support anymore) but mentioned that earlier versions of PnP worked, can you give us the version where you didn't have the issue? Was it 1.9?

[JazBInKC]

As you seem to use SharePoint on-prem (that we don't support anymore) but mentioned that earlier versions of PnP worked, can you give us the version where you didn't have the issue? Was it 1.9?

Not sure why you think I'm on-prem, I'm actually downloading (and continually updating) 15 million files from SharePoint Online to NTFS. I had originally started this project with SharePointPnP (version 3 something) and then updated to 1.10.0 in a effort to resolve other issues (with memory usage on Get-PnPListItem). Anyway, I hadn't noticed this particular issue at that time. I've been upgrading to the nightly releases - again because of other issues cropping up with PS 5.1 / PS 7 compatibility, etc. so not exactly sure when this stopped working correctly - my code has not changed. And, while I don't encourage the use of "+", I don't see where it's not allowed in folder and file names or that it's considered an escape character as used. It's unclear why the "+" is replaced by " " in the error message path when the $serverrelativeurl = $item.FieldValues.FileRef value appears to be correct.

Hoping someone can fix this. I'd rather not have to test millions of files / folders for "+" in the path.

jbw

[milanholemans]

The problem is that the specified URL is always decoded within the cmdlet by this piece of code:

https://github.com/pnp/powershell/blob/f2c41de2fbc6defce6e6cc304b184c3c2a776b8c/src/Commands/Files/GetFile.cs#L71-L72

Because a + sign is the encoded variant of a space, a path named one+two will become one two. Since this path is not valid, it throws a not found error. I'm not sure if we can provide a clean fix for this since it is quite hard to tell whether a provided URL is encoded or decoded.

What you can do to make your script work is pretty easy. If you encode all your URLs, your script will work just fine. Simply do something like this:

$encodedUrl = [System.Web.HttpUtility]::UrlEncode("/sites/HR/Shared Documents/Folder one+two/Document.pdf")
Get-PnPFile -ServerRelativeUrl $encodedUrl

[JazBInKC]

I appreciate the suggestion. It's disturbing that code that worked previously no longer works due to some "improvement". Earlier versions of PnP.PowerShell treated the SharePoint ServerRelativeUrl as a literal path.

Thanks!

[KoenZomers]

Interesting one. I know we've had reports before where special characters wouldn't work. That was fixed by adding this encoding. As @milanholemans states above, it's now juggling between accepting the plusses breaking the öä and other characters or having those work and the plusses error out. Not sure what's wisdom here. What if we explicitly code out the plusses before we encode?

[KoenZomers]

Got a repro here. Let me see what I can do to fix both scenarios.

[KoenZomers]

Got a fix by simply URL encoding any plusses in the URL. Hopefully that's the only odd character that's problematic here. Let me build a PR for it so we can get it into our nightly builds so you can test it.

[gautamdsheth]

Closed, will be available in tomorrow's nightly.

[JazBInKC]

I’ve been encoding the paths prior to Get-Pnpfile as my workaround. Can’t we just add -literalpath to determine if the path should be encoded or not?

jbw

Sent via causality loop 3 days from now.

On Jun 16, 2022, at 6:52 AM, Koen Zomers @.***> wrote:

 Attention: This is an external email. Please do not click links or open attachments unless you recognize the source of this email and know the content is safe.

Got a fix by simply URL encoding any plusses in the URL. Hopefully that's the only odd character that's problematic here. Let me build a PR for it so we can get it into our nightly builds so you can test it.

— Reply to this email directly, view it on GitHubhttps://github.com/pnp/powershell/issues/1864#issuecomment-1157570911, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AW7N6ARFD2LDJPH7SGFBHG3VPMIPDANCNFSM5V5AKEMA. You are receiving this because you were mentioned.Message ID: @.***>

[milanholemans]

Actually, I don't understand why we expect the user to provide an encoded URL. Why was this added?

[KoenZomers]

The Get-PnPFile is a bit of a mess already with all the different variants it tries to support, not really in favor of adding yet another parameter to it. What's the problem with doing it this way? If you already encode them yourself, it should still work fine, no?

@milanholemans, can't remember the exact scenario, but I remember we had a couple of people reporting issues on it that lead to the decoding thing. Perhaps if you copy/copied the URL or a file somewhere it was encoded? Can't remember.

[diegoweb100]

same error if you try to save file to local path...

[diegoweb100]

...trying to help ...Version 1.7.0 is working...

[myfriendedward]

If you already encode them yourself, it should still work fine, no?

Actually, I'm finding that no, it does not work fine if I encode them myself.

For example, if I have a literal path of one two, that gets encoded to one+two when I run it through [System.Web.HttpUtility]::UrlEncode. Then, when I pass that encoded URL to Get-PnPFile, the plus no longer gets decoded back to a space due to the special treatment of plusses, which leads to a "file does not exist" error because it is looking for a file with a plus instead of a space.

When I was on version 1.10.0, I had added the call to UrlEncode because files with plusses in their name would fail if they were not encoded.

Now, in version 1.12.0, I am removing those same calls to UrlEncode because files with spaces in their name are failing when they are encoded.

[KoenZomers]

Thanks for your addition @myfriendedward . It would tremendously help if people running into issues with Get-PnPFile can provide concrete copy/paste Get-PnPFile samples with a short description with them what goes wrong and what was expected so I can try to see what we can do to deal with the many scenarios that we have. Fixing one scenario seems to wreck it for other scenarios.

[myfriendedward]

Thanks for reopening, @KoenZomers .

There are situations where Get-PnPFile will fail regardless of whether the file name is encoded or not, at least when the encoding uses plus for spaces instead of %20.

For example, the literal name %2begin v2 will fail either way:

• When not encoded, the literal %2b is decoded to a plus.
• When encoded using [System.Web.HttpUtility]::UrlEncode, the space becomes a plus that does not get decoded back to a space.
• However, when encoded using [System.Uri]::EscapeDataString, it works because the space is encoded as %20 instead of plus.

Code for this example:

$rootFold = '/sites/YourSite/YourFolder/'
$filename = '%2begin v2'

# Without encoding, leads to error "The file /sites/.../+egin v2 does not exist."
# because the literal %2b is decoded to a plus.
$url = $rootFold + $filename
Get-PnPFile -Url $url -Path 'C:\temp' -Filename $filename -AsFile -Force    # FAILS


# With encoding via [System.Web.HttpUtility]::UrlEncode, leads to
# error "The file /sites/.../%2begin+v2 does not exist."
# This outcome is due to [System.Web.HttpUtility]::UrlEncode using a
# plus instead of %20 as the encoded value for a space, followed by
# Get-PnPFile treating plus as a special character that does not get decoded.
$url = [System.Web.HttpUtility]::UrlEncode($rootFold + $filename)
Get-PnPFile -Url $url -Path 'C:\temp' -Filename $filename -AsFile -Force    # FAILS


# With encoding via [System.Uri]::EscapeDataString, works correctly
# because the space is encoded as %20 instead of plus.
$url = [System.Uri]::EscapeDataString($rootFold + $filename)
Get-PnPFile -Url $url -Path 'C:\temp' -Filename $filename -AsFile -Force    # WORKS

Based on the above, I would conclude that currently the only "safe" way to pass a file name to Get-PnPFile is to encode it first, and the encoding must use %20 rather than plus for spaces.

[JazBInKC]

What's missing here is -LiteralPath. It used to be a feature of most PnP calls. With the later 1.2x versions of PnpFile I have stopped pre-encoding. "+" logic was included in the function (1.14??), encoding screws that up.

jbw

[myfriendedward]

@JazBInKC , agreed, a -LiteralPath parameter seems like a very clean way to resolve the issue.

As an extension of the example I had posted above, there are scenarios where, counterintuitively, it is necessary to encode the ServerRelativeUrl returned from Get-PnPFolderItem before that value can be passed to Get-PnPFile:

$relativeRoot = 'YourFolder'  # Folder containing file with literal name '%2begin v2'

$files = @(Get-PnPFolderItem -FolderSiteRelativeUrl $relativeRoot -ItemType File)

# Passing the file's ServerRelativeUrl as returned from Get-PnPFolderItem
# to Get-PnPFile can fail, because Get-PnPFolderItem does NOT encode the URL,
# whereas Get-PnPFile treats it as an encoded URL (with a special exception for plusses).
# Error in this case is "The file /sites/.../+egin v2 does not exist", because the
# literal '%2b' in the name was incorrectly interpreted as an encoded plus.
foreach ($file in $files) {
	Get-PnPFile -Url $file.ServerRelativeUrl -Path 'C:\temp' -Filename $file.Name -AsFile -Force    # FAILS
}


# Must encode the URL that was returned from Get-PnPFolderItem
# before it can safely be passed to Get-PnPFile.
foreach ($file in $files) {
	Get-PnPFile -Url ([System.Uri]::EscapeDataString($file.ServerRelativeUrl)) -Path 'C:\temp' -Filename $file.Name -AsFile -Force    # WORKS
}

Given that Get-PnPFolderItem returns non-encoded URLs, it seems like the default behavior of Get-PnPFile should be to assume non-encoded URLs as well.

[Aurelius20k]

I'm having likewise decoding issues with Get-PnPFile where a -LiteralPath would definitely help. For me the issue is with %20 (encoded space). Yes, I know why would you put that in a filename in SharePoint but I am working with large archives where stuff just gets uploaded (dumped) into and end users don't really know this causes issues.

Done some testing:

Original URL, get Get-PnPFile replaces %20 in the filename with a space:

Input: 'Test%20Test.pdf'

$Url = "/sites/somesitecollection/somesubsite/someothersubsite/somelibrary/somefolder/Test%20Test.pdf"
Get-PnPFile -url $Url -Path "C:\Temp" -Filename "Test%20Test.pdf" -AsFile
The file /sites/somesitecollection/somesubsite/someothersubsite/somelibrary/somefolder/Test Test.pdf does not exist.

Result: 'Test Test.pdf'

Okay, that does not work, why don't I add %25 (%) to the URL so it decodes to %20:

Input: 'Test%2520Test.pdf'

$Url = "/sites/somesitecollection/somesubsite/someothersubsite/somelibrary/somefolder/Test%2520Test.pdf"
Get-PnPFile -url $Url -Path "C:\Temp" -Filename "Test%20Test.pdf" -AsFile
The file /sites/somesitecollection/somesubsite/someothersubsite/somelibrary/somefolder/Test Test.pdf does not exist.

Result: 'Test Test.pdf'

That also didn't work, the space is still added and no % is present. Adding %2525 for testing just adds one % and still adds the space:

Input: 'Test%252520Test.pdf'

$Url = "/sites/somesitecollection/somesubsite/someothersubsite/somelibrary/somefolder/Test%252520Test.pdf"
Get-PnPFile -url $Url -Path "C:\Temp" -Filename "Test%20Test.pdf" -AsFile
The file /sites/somesitecollection/somesubsite/someothersubsite/somelibrary/somefolder/Test% Test.pdf does not exist.

Result: 'Test% Test.pdf'

Coding the URL with stuff like [System.Web.HttpUtility]::UrlEncode and [System.Uri]::EscapeDataString also does not work.

Seeing 'Test%252520Test.pdf' gets decoded to 'Test% Test.pdf' makes it seem like there is some kind of double decoding going on here. It decodes '%252520' to '%%2520' and then to '% ' (with space).

Edit, I'm on PnP.PowerShell 2.3.0, PowerShell 7.

[mdurini-swisscom]

i'm facing the same problem, PnP.PowerShell 2.3.0, PowerShell 7. i was able to get it working partially with the workaround [System.Web.HttpUtility]::UrlEncode but if the filename contains %20, it's not working....

[andreyp115]

Same problem. The only option I found for files with %20 is to rename at the source. Please provide LiteralPath parameter! PowerShell 7.4.1 PnP.PowerShell 2.4.0

[mycodeisloud]

Same here.