Allow setting all headers for Invoke-WebRquest

[Bartolomeus-649]

Invoke-WebRequest should not need a session to specify cookies

Today you need to use a session to be able to specify cookies for a request with Invoke-WebRequest, that should not be needed.

This restriction that you need to use a session to include cookies in a request makes it very complicated to "replay" recorded requests, which is a great help when you need to reproduce tricky problems where you need to reach a specific state. As an example, if you use the new Chromium-based Edge, and copies a request from the network trace in the devtools as PowerShell, it won't work, since all headers, including cookies and useragent are specified in the -Headers parameter.

Proposed technical implementation details (optional)

Start by using/setting all headers as specified with the -Headers parameter, then if Sessions and/or -UserAgent are used, then just use them instead and overwrite any previous headers.

And this is what the PowerShell Edge produce looks like:

Invoke-WebRequest -Uri "https://abc.go.com/" -Headers @{"method"="GET"; "authority"="abc.go.com"; "scheme"="https"; "path"="/"; "pragma"="no-cache"; "cache-control"="no-cache"; "upgrade-insecure-requests"="1"; "user-agent"="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.18 Safari/537.36 Edg/75.0.139.4"; "dnt"="1"; "accept"="text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3"; "accept-encoding"="gzip, deflate, br"; "accept-language"="en-US,en;q=0.9,"; "cookie"="SWID=2727B4CB-A7BB-41CC-AB3D-7C9BC01CF4D7"}

[markekraus]

Method, scheme, and path are not usually headers. Those are part of the HTTP start line and not the headers proper.

Even taking that into account... The example you provided works without issue:

$Headers = @{
    "method" = "GET"
    "authority" = "abc.go.com"
    "scheme" = "https"
    "path" = "/"
    "pragma" = "no-cache"
    "cache-control" = "no-cache"
    "upgrade-insecure-requests" = "1"
    "user-agent" = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.18 Safari/537.36 Edg/75.0.139.4"
    "dnt" = "1"
    "accept" = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3"
    "accept-encoding" = "gzip, deflate, br"
    "accept-language" = "en-US,en;q=0.9,"
    "cookie" = "SWID=2727B4CB-A7BB-41CC-AB3D-7C9BC01CF4D7"
}
$Uri = 'https://httpbin.org/get'
$Result = Invoke-WebRequest -Uri $Uri -Headers $Headers
$ResponseObject = $Result.Content | ConvertFrom-Json

foreach($Item in $Headers.GetEnumerator()){
    $Key = $Item.Key
    if (-not ($ResponseObject.headers.$Key -eq $Item.Value)){
        'Key: {0}' -f $Key
        'Headers: {0}' -f $Item.Value
        'Response: {0}' -f $ResponseObject.headers.$Key
        '---'
    }
}

Result:

Key: accept-language
Headers: en-US,en;q=0.9,
Response: en-US, en; q=0.9
---
Key: accept
Headers: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
Response: text/html, application/xhtml+xml, application/xml; q=0.9, image/webp, image/apng, */*; q=0.8, application/signed-exchange; v=b3
---

And those differences only arise because the HttpClient API automatically cleans up dirty headers.

[markekraus]

For clarity, sessions are not required for cookies and can definitely be sent in the Headers param.

[Bartolomeus-649]

From the documentation

-Headers

Specifies the headers of the web request. Enter a hash table or dictionary.

To set UserAgent headers, use the -UserAgent parameter. You cannot use this parameter to specify User-Agent or cookie headers.

And no, it does not work. This command: Invoke-WebRequest -Uri "http://www.microsoft.com" -Headers @{"cookie" = "MyCookieVarable=MyCookieValue"}

Produce this request as caught by Fiddler

GET http://www.microsoft.com/ HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT; Windows NT 10.0; sv-SE) WindowsPowerShell/5.1.17134.590
Host: www.microsoft.com
Connection: Keep-Alive

As you can see, there are no cookies sent.

[markekraus]

what version of PowerShell?

[Bartolomeus-649]

10.0.17134.1

[markekraus]

That looks to be a PowerShell 5.1 issue. Please verify the issue still exists in 6.2.0.

[Bartolomeus-649]

ok, seems to work better in 6.2.0, but these version numbers seems a bit confusing, the first one clearly say it is version 10.0.

This is what the same request look like according to Fiddler in 6.2, which is much beter

GET http://www.microsoft.com/ HTTP/1.1
Cookie: MyCookieVarable=MyCookieValue
User-Agent: Mozilla/5.0 (Windows NT 10.0; Microsoft Windows 10.0.17134; sv-SE) PowerShell/6.2.0
Host: www.microsoft.com

and, in the doc for 6.2 it still say:

-Headers <IDictionary> Specifies the headers of the web request. Enter a hash table or dictionary.

To set UserAgent headers, use the -UserAgent parameter. You cannot use this parameter to specify User-Agent or cookie headers.

[markekraus]

I recommend opening an issue in the docs repo or submitting a PR to fix it in the 6+ docs. https://github.com/MicrosoftDocs/PowerShell-Docs/issues/new

[CarloToso]

@iSazonov I think we can close this issue

[iSazonov]

We need to open new issue in Docs repo.

[CarloToso]

@iSazonov fixed https://github.com/MicrosoftDocs/PowerShell-Docs/pull/9757