Kestrel Exception on urlencoded paths when Content Negotiation adds links to responses

[csainty]

Prerequisites

• [x] I have written a descriptive issue title
• [x] I have verified that I am running the latest version of Nancy
• [x] I have verified if the problem exist in both DEBUG and RELEASE mode
• [x] I have searched open and closed issues to ensure it has not already been reported

Description

When the path has urlencoded characters in it and content negotiation attempts to add link headers to the response. It does so in a manner that causes Kestrel to throw

System.InvalidOperationException: Invalid non-ASCII or control character in header: 0x00E5
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.FrameHeaders.ThrowInvalidHeaderCharacter(Char ch)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.FrameHeaders.ValidateHeaderCharacters(String headerCharacters)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.FrameHeaders.ValidateHeaderCharacters(StringValues headerValues)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.FrameResponseHeaders.SetValueFast(String key, StringValues value)
   at Nancy.Owin.NancyMiddleware.RequestComplete(NancyContext context, IDictionary`2 environment, Func`2 performPassThrough, Func`2 next)

This happens because the characters that were encoded in the original url are passed unencoded to the response header, violating HTTP spec.

Steps to Reproduce

1. Setup a plain Nancy project with Kestrel as the host
2. Create a route Get("/{slug}", _ => new { Test = "Hello World"});
3. curl -v http://localhost:5000/george-%C3%A5

System Configuration

• Nancy version: 2.0.0-clinteastwood (also present in barney, under slightly different trigger)
• Nancy host
  • [ ] ASP.NET
  • [ ] OWIN
  • [ ] Self-Hosted
  • [x] Other: Kestrel
• Other Nancy packages and versions:
• Environment (Operating system, version and so on): OSX
• .NET Framework version: 1.0.0-preview2-1-003177
• Additional information:

As a workaround, you can return Response.AsJson() which bypasses the code to add links e.g. Get("/foo/{slug}", _ => Response.AsJson(new { Test = "Hello World"})); See https://github.com/NancyFx/Nancy/blob/master/src/Nancy/Responses/Negotiation/DefaultResponseNegotiator.cs#L42 for the bypass

By encoding a character that does not cause Kestrel to crash, you can see how it ends up unencoded in the response

$ curl -v http://localhost:5000/george-%3B
> GET /george-%3B HTTP/1.1
> Host: localhost:5000
> User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)
> Accept: */*
> Referer:
>
< HTTP/1.1 200 OK
< Date: Mon, 30 Jan 2017 18:56:46 GMT
< Content-Type: application/json; charset=utf-8
< Server: Kestrel
< Transfer-Encoding: chunked
< Vary: Accept
< Link: </george-;.xml>; rel="alternate"; type="application/xml"
<
{"Test":"Hello World"}

[khellang]

Closed with #2903