Http request with header transfer-encoding set to chunked is not handled as expected

[soninaren]

Function runtime: v2 Environment: Core-tools and Azure

Repro steps:

1. Create a function app
2. Create a function using httpTrigger template
3. Send a http request with encoding set to chunked via the following this console app: repro

It works as expected for C# and csx functions apps. but for node the function runtime processes the request without the request body which results in a 400 Bad request.

[dkovela]

This is basically blocking the proposed design and implementation for an important project. If this is not fixed, we will have to re-design and implement the planned integration. Can you please fix this as priority ASAP or please let us know ETA if you have it.

[crklein]

Can we please get an update? Our project is in day by day slip without this resolved.

[soninaren]

cc: @fabiocav , @ColbyTresness

[ColbyTresness]

@mhoeger as well

[crklein]

We are still stuck and haven't heard anything regarding this issue. Can we please get a status update?

[joain946]

I'm experiencing the same issue except that I'm running with Java locally. If I set a Content-Length then it works. But of course that should not be needed for a chunked transfer. Any plan to create a correction for this, since it is a year old ticket?

My environment: Java 11 Azure Functions Core Tools (3.0.2798 Commit hash: d9d99872a51c04b1c2783198b1ee1104da6b064f) Function Runtime Version: 3.0.14191.0

[rkaw92]

Having the same problem with a Node.js function - a POST with chunked encoding results in both req.body and req.rawBody being undefined, so there doesn't seem to be a way to actually access the data. This is on a local environment started with func start - tested with both Functions 2.0 and the Functions 3.0 runtimes.

[fabiocav]

@soninaren (and others on the thread), there was an update recently made to the HTTP binding that could potentially help with this, so we need to validate whether that is indeed the case.

This is an opt-in behavior (as it does have security implications). In order to enable to feature, you need to update the binding configuration in host.json with the following:

{
    "extensions": {
        "http": {
            "EnableChunkedRequestBinding": true
          }
    }
}

[fabiocav]

Triaging for assignment (for validation only)

[fabiocav]

@soninaren could you also follow up with a documentation issue to make sure this is publicly documented (with the warnings). @alrod should be able to assist here as well.

[tom-odon]

@soninaren (and others on the thread), there was an update recently made to the HTTP binding that could potentially help with this, so we need to validate whether that is indeed the case.

This is an opt-in behavior (as it does have security implications). In order to enable to feature, you need to update the binding configuration in host.json with the following:

{
    "extensions": {
        "http": {
            "EnableChunkedRequestBinding": true
          }
    }
}

@fabiocav Can you detail more about what the security implications are?

[dvdgutzmann]

Hi!

{
    "extensions": {
        "http": {
            "EnableChunkedRequestBinding": true
          }
    }
}

@fabiocav Was this improvement only made to http binding for the v2 Function runtime or is this extensions configuration also available for v3 and v4? What about in-process vs. isolated? I'm unable to find any documentation on this config option.

We' are currently blocked by this with a dotnet-isolated function running .NET 5 on runtime v3. See Azure/azure-functions-dotnet-worker#713

Thanks! David

[dvdgutzmann]

Hi!

{ "extensions": { "http": { "EnableChunkedRequestBinding": true } } }

@fabiocav Was this improvement only made to http binding for the v2 Function runtime or is this extensions configuration also available for v3 and v4? What about in-process vs. isolated? I'm unable to find any documentation on this config option.

I just did a quick test with

curl -ivsS --trace -H "Transfer-Encoding: chunked" --data 123 -X POST http://localhost:7071/api/fn

and this setups

• v3 runtime, dotnet, .NET Core 3.1
• v3 runtime, dotnet-isolated, .NET 5
• v4 runtime, dotnet, .NET 6
• v4 runtime, dotnet-isolated, .NET 6

Observations:

1. The EnableChunkedRequestBinding option has no effect at all in v3 and v4 setups.
2. The functions running out-of-process with "FUNCTIONS_WORKER_RUNTIME": "dotnet-isolated" pass on NullStreams to the functions - therefore the request body is NOT accessible by the function.
3. The functions running in-process with "FUNCTIONS_WORKER_RUNTIME": "dotnet" behave like all of us expect them to do - the request body is accessible by the function.

Maybe I'm going out on a limb but I do consider the mishandling of chunked transfers in workers running out-of-process a bug - especially when none of it is documented anywhere.

@fabiocav Do you agree with my analysis? Even if you don't concur with my conclusion, did I get the facts straight?

Thanks! David

[Zhaph]

Just to add more keywords here:

This issue means that you cannot send requests to an HTTP Trigger on the function runtime using the HttpClient PostAsJsonAsync extension method as per https://github.com/aspnet/AspNetWebStack/issues/252 and instead need to manually create a StringContent body when sending JSON requests to a Function App.

This is considered a "server side fix and issue".

[mrurb]

This is still an ongoing issue. Is there any progress to fix this issue? We were looking into upgrading our function to .Net6 isolated, but this is currently stopping us.

[Rashair]

Same issue with Python - Content-Length header is required, otherwise the request is not parsed.

[MitchBodmer]

I feel like it's important to stress here again that this bug in combination with the fact that PostAsJsonAsync and PostAsync with JsonContent do not set the Content-Length means you can't post JSON to an isolated HTTP Azure Function without a non-obvious workaround. This also still does not seem to be documented anywhere, but I may have missed something. It also presents itself in a confusing way: as a completely missing request body with no warning.

I really can't think of a use case for an HTTP Triggered Azure Function that I've seen more often than "post JSON to it from a C# application", so I'm confused why this issue has languished for three years. Now that we're five days away from developers not being able to migrate to .net 7 functions unless they also move to isolated functions, this seems especially urgent.

@fabiocav am I missing something here?

[MitchBodmer]

#https://github.com/Azure/azure-functions-dotnet-worker/issues/374

Here is another issue where a number of developers wrestled with this issue and never got to the root cause, though they did find the workaround.

[yggdrasil-tynor]

.NET 7 released and still not possible to upgrade to isolated functions due to this bug?

[imhauser]

I am considering stopping using Azure Functions in our projects. It has been a nightmare to upgrade to .net5 and now .net7. Our functions are called by SAP and I no control on the transfer-encoding parameter. (chunked). So when chunked, body is empty in isolated function (.net7) I am surprise there is not more reports about this problem

[MitchBodmer]

I don't know if there's been any progress on this, but I'll say again that it seems quite bad. Since it isn't documented and has such a confusing and seemingly unrelated presentation, I would hazard a guess that there are far more people encountering it than the activity on this bug would suggest.

[justin-appointmentreminder]

Just hit this issue myself with a .NET 6 Isolated Function (C# v4 runtime). My client webhook caller is using Transfer-Encoding: Chunked, and the body in the function is null. Very very frustrating. Starting to get concerned if Microsoft Technologies are the right way to go.

[mrurb]

@cloudmelon I was made aware that you might be able to help us with this issue. It is currently stopping our and other companies from upgrading to Isolated Workers.

[cloudmelon]

Thanks, @mrurb we'll take a look at this and get back to you.

[cloudmelon]

Hi @mrurb i checked with the team, and this is an opt-in behavior that's not enabled by default due to the security implications. In order to enable to feature, you need to update the binding configuration in host.json with the following:

{
    "extensions": {
        "http": {
            "EnableChunkedRequestBinding": true
          }
    }
}

And this should be able to resolve your issue, let us know if that works and hopefully unblock your customers. I am tagging @fabiocav here for vis.

[mrurb]

Hi @mrurb i checked with the team, and this is an opt-in behavior that's not enabled by default due to the security implications. In order to enable to feature, you need to update the binding configuration in host.json with the following:

{
    "extensions": {
        "http": {
            "EnableChunkedRequestBinding": true
          }
    }
}

And this should be able to resolve your issue, let us know if that works and hopefully unblock your customers. I am tagging @fabiocav here for vis.

Hi @cloudmelon as mentioned earlier in this thread, the configuration in the host.json seems to have no effect.

Hi!

{ "extensions": { "http": { "EnableChunkedRequestBinding": true } } }

@fabiocav Was this improvement only made to http binding for the v2 Function runtime or is this extensions configuration also available for v3 and v4? What about in-process vs. isolated? I'm unable to find any documentation on this config option.

I just did a quick test with

curl -ivsS --trace -H "Transfer-Encoding: chunked" --data 123 -X POST http://localhost:7071/api/fn

and this setups

• v3 runtime, dotnet, .NET Core 3.1
• v3 runtime, dotnet-isolated, .NET 5
• v4 runtime, dotnet, .NET 6
• v4 runtime, dotnet-isolated, .NET 6

Observations:

1. The EnableChunkedRequestBinding option has no effect at all in v3 and v4 setups.
2. The functions running out-of-process with "FUNCTIONS_WORKER_RUNTIME": "dotnet-isolated" pass on NullStreams to the functions - therefore the request body is NOT accessible by the function.
3. The functions running in-process with "FUNCTIONS_WORKER_RUNTIME": "dotnet" behave like all of us expect them to do - the request body is accessible by the function.

Maybe I'm going out on a limb but I do consider the mishandling of chunked transfers in workers running out-of-process a bug - especially when none of it is documented anywhere.

@fabiocav Do you agree with my analysis? Even if you don't concur with my conclusion, did I get the facts straight?

Thanks! David

Just to make sure that anything has not changed since last tested, I just tested it and it still has no effect: with v4 runtime and dotnet-isolated and EnableChunkedRequestBinding set to true any chunked request still returns a null stream:

Running a request without chunked works fine:

[yggdrasil-tynor]

@cloudmelon @fabiocav We really need some clarification around this bug. I hope that this will eventually be fixed, but if there are no plans you should really let the community know so we can settle on other solutions. This was opened 3.5 years ago!! 👎

[cloudmelon]

Hi @mrurb i checked with the team, and this is an opt-in behavior that's not enabled by default due to the security implications. In order to enable to feature, you need to update the binding configuration in host.json with the following:

{
    "extensions": {
        "http": {
            "EnableChunkedRequestBinding": true
          }
    }
}

And this should be able to resolve your issue, let us know if that works and hopefully unblock your customers. I am tagging @fabiocav here for vis.

Hi @cloudmelon as mentioned earlier in this thread, the configuration in the host.json seems to have no effect.

Hi!

{ "extensions": { "http": { "EnableChunkedRequestBinding": true } } }

@fabiocav Was this improvement only made to http binding for the v2 Function runtime or is this extensions configuration also available for v3 and v4? What about in-process vs. isolated? I'm unable to find any documentation on this config option.

I just did a quick test with

curl -ivsS --trace -H "Transfer-Encoding: chunked" --data 123 -X POST http://localhost:7071/api/fn

and this setups

• v3 runtime, dotnet, .NET Core 3.1
• v3 runtime, dotnet-isolated, .NET 5
• v4 runtime, dotnet, .NET 6
• v4 runtime, dotnet-isolated, .NET 6

Observations:

1. The EnableChunkedRequestBinding option has no effect at all in v3 and v4 setups.
2. The functions running out-of-process with "FUNCTIONS_WORKER_RUNTIME": "dotnet-isolated" pass on NullStreams to the functions - therefore the request body is NOT accessible by the function.
3. The functions running in-process with "FUNCTIONS_WORKER_RUNTIME": "dotnet" behave like all of us expect them to do - the request body is accessible by the function.

Maybe I'm going out on a limb but I do consider the mishandling of chunked transfers in workers running out-of-process a bug - especially when none of it is documented anywhere. @fabiocav Do you agree with my analysis? Even if you don't concur with my conclusion, did I get the facts straight? Thanks! David

Just to make sure that anything has not changed since last tested, I just tested it and it still has no effect: with v4 runtime and dotnet-isolated and EnableChunkedRequestBinding set to true any chunked request still returns a null stream:

Running a request without chunked works fine:

The current HTTP model for isolated doesn't all head chunking scenarios, but we're working on a new design that will help resolve related challenges here

[mattchenderson]

Confirmed that EnableChunkedRequestBinding does not seem to work. We need to retriage this.

In addition to fixing the behavior, part of the work here is to ensure that this gets properly documented.

[MitchBodmer]

@fabiocav @mattchenderson Is there a reason this was moved out of Sprint 147, but not into any other sprint? I was looking forward to this being addressed.