HeaderPropagation middleware: MessageHandler throws when used outside of a request

[alefranz]

trafficstars

Describe the bug

The MessageHandler relies on the Middleware to initialize HeaderPropagationValues.Headers to detect misconfiguration. However this means the Messagehandler can not be used outside of an http request and currently throws:

System.InvalidOperationException: The HeaderPropagationValues.Headers property has not been initialized. If using this HttpClient as part of an http request, register the header propagation middleware by adding 'app.UseHeaderPropagation() in the 'Configure(...)' method. Otherwise, use HeaderPropagationProcessor.ProcessRequest() before using the HttpClient.

To Reproduce

Steps to reproduce the behavior:

Code example

public void ConfigureServices(IServiceCollection services)
{
    services
        .AddHttpClient("test")
        .AddHeaderPropagation();
    services.AddHostedService<SampleHostedService>();
}

public class SampleHostedService : IHostedService
{
    private readonly IHttpClientFactory _httpClientFactory;
    private readonly HeaderPropagationProcessor _headerPropagationProcessor;
    private readonly ILogger _logger;

    public SampleHostedService(IHttpClientFactory httpClientFactory, HeaderPropagationProcessor headerPropagationProcessor, ILogger<SampleHostedService> logger)
    {
        _httpClientFactory = httpClientFactory ?? throw new ArgumentNullException(nameof(httpClientFactory));
        _headerPropagationProcessor = headerPropagationProcessor ?? throw new ArgumentNullException(nameof(headerPropagationProcessor));
        _logger = logger ?? throw new ArgumentNullException(nameof(logger));
    }

    public Task StartAsync(CancellationToken cancellationToken)
    {
        return DoWorkAsync();
    }

    private async Task DoWorkAsync()
    {
        _logger.LogInformation("Background Service is working.");

        var client = _httpClientFactory.CreateClient("test");
        var result = await client.GetAsync("http://localhost:62013/forwarded");

        _logger.LogInformation("Background Service:\n{result}", result);
    }

    public Task StopAsync(CancellationToken cancellationToken)
    {
        return Task.CompletedTask;
    }
}

Expected behavior

The HttpClient should work if used outside of a request or, if we don't wont to support this use case, the exception should be updated to explain that such use case is not supported..

Possible solutions

I had investigated possible solutions. Not sure what is your preferred options and if we are still in time to do breaking changes on this middleware. Here are some options:

1. Remove the check that throws the exception which means we would not be able anymore to detect misconfiguration. Can this be replaced with an analyzer? On top of the risk that the feature might not work due to misconfiguration, there is also the risk that having the valuefilter not execute (e.g. to set some headers with default values) when not called from outside a request can be unexpected.
2. Update the exception message to also explain that a HttpClient with the HeaderPropagationMessageHandler can not be used outside of a request. This is not ideal as it means you end up duplicating your HttpClients configuration.
3. Extract the functionality from the Middleware to a generic Processor that can be called outside a http request. This will not change the middleware behaviour, but will allow to explicitly call the Processor when consuming an HttpClient outside of a http request. Ideally this will also include changing the valueFactory to receive a headers collection instead of the HttpContext, so that headers can be passed in different scenarios, for example when consuming messages from a queue, however I guess this scenario is out of scope for a middleware. See draft PR #12170 . The main issue I see with this approach is that when the processor is used directly you must take extra care as you need a different async context per request.

Looking forward for your feedback.

Thank you, Alessio

/cc @rynowak

[rynowak]

The HttpClient should work if used outside of a request or, if we don't wont to support this use case, the exception should be updated to explain that such use case is not supported..

Can you describe what this would even mean to "support this use case". Is the idea that you want to use header propagation where possible and then degrade gracefully?

I guess I'm more interested to hear about the scenario than the technical changes.

[alefranz]

Hi Ryan,

Is the idea that you want to use header propagation where possible and then degrade gracefully? that is what I meant. Apologies if my explanation was a bit convoluted.

I believe having some background processing in an API is not so uncommon. And probably that background process will deal with the same dependent service that some controller's action deal with directly.

When a user add the HeaderPropagation behavior to a client, does not probably realize that this means that the client can not be used anymore outside of a incoming HTTP request transaction (as we rely on the middleware initializing the headers collection to detect misconfiguration). So there is a requirement of making this constraint more obvious.

Unfortunately when using a typed client and relying on DI, it become convoluted to have to exact same clients just with a different configuration, so I am not sure that this behavior is acceptable.

The alternative that consuming the same client from a hosted service is made possible, but without propagating the headers, to me feels more user friendly, however there could be a scenario when the background processing is triggered by some incoming http request (e.g. storing some data on a queue for later processing by a hosted service in the same process) and a user will incorrectly assume that the headers are propagated and doesn't get any exception to warn about it. So in this respect the current behavior is more safe, although can be a bit annoying to deal with it.

Does this answer your question or are you looking for an example?

I don't have a real example of a service with such scenario (otherwise I would have caught this before 😅) but it could be a service when you get a request that involve a minimum call to a service A to get the data required to fulfill the response, and it defer to later some more expensive call to service A which is a consequence of the original request but not required to return the response (as it is only a side effect or it will be sent back with a webhook). This could be for example achieved using a a TPL Dataflow ActionBlock or a queuing mechanism than consumed by a hosted service. The key part is that the processor will have to consume the same service A.

I can put together an example if that will help.

Thank you, Alessio

[rynowak]

Extract the functionality from the Middleware to a generic Processor that can be called outside a http request. This will not change the middleware behaviour, but will allow to explicitly call the Processor when consuming an HttpClient outside of a http request. Ideally this will also include changing the valueFactory to receive a headers collection instead of the HttpContext, so that headers can be passed in different scenarios, for example when consuming messages from a queue, however I guess this scenario is out of scope for a middleware.

I don't really want to do this I guess (anymore so than we already did). The idea of adding this feature to the core was that getting the capture + propagation of values from the current request is a tricky problem that seemed worth ASP.NET Core solving for you.

Writing a general purpose run code to set headers feature is too general, and something that anyone can do with a message handler in whichever way they please.

Put another way: it's worth us providing opinions about how to integrate features of ASP.NET Core with each other, but it's not a great idea for us to provide opinions about all the code you write. It's better to just tell people to use the BCL features than to provide wrappers around them - for us to provide an alternative to some code you could write - there has to be some kind of value to it other than reduction in lines of code

---

I think the scenario makes sense. It's a little surprising that once you opt-in to header forwarding, you can't use that client outside of a request - but it's also a reasonable explanation - but the workaround is painful (duplicate your config)

I think are options are to either:

• Do nothing (can always be fixed in the future)
• Remove the sanity check
• Add an option to bypass the sanity check

We have a few days left to address this before the release really locks down - I've taken a while to respond because I'm not sure which of the last two options is least surprising.

@anurse @Tratcher @davidfowl @glennc

[analogrelay]

We have a few days left to address this before the release really locks down

Barely that even. I think this might be a good candidate to reconsider when we open a branch for 3.1 next week. I'm afraid I haven't been able to catch up on the thread, but is this something that will break customer scenarios if not resolved? If so, we can evaluate it for 3.0 (based on the impact of the bug). If this is just about expanding the usefulness of the HeaderPropagation message handler, I think we'll have to move this out of 3.0. It wouldn't meet the bar.

[rynowak]

There's no break here.

We're discussing how to best deal with a case that would throw an exception in 3.0 - and trying to decide if we want to make it more tolerant. I have no issue with moving this decision to 3.1

[alefranz]

Yes I agree we need to postpone it as we are waiting for more feedback on what is the best approach to take.

@rynowak do we need to rephrase the exception message for 3.0? we could add something at the end to indicate that another possible reason of this exception is that the Http client is used outside of processing of a request which is unsupported.

Thanks!

[rynowak]

Sure, if you want to send a PR that updates the exception message that would be fine. We have until monday to merge it.

[alefranz]

Raised PR https://github.com/aspnet/AspNetCore/pull/12636

[raphaabreu]

Hello all, I am coming here from https://github.com/alefranz/HeaderPropagation/issues/6 as suggested by @alefranz.

I am starting to upgrade some projects at my company and header forwarding is being an issue for us. It is awesome that there is a standard package from the AspNetCore team that we can reference and use as model to how to solve similar problems, however this particular implementation is breaking for us. The header forwarding functionality should not break the regular httpclient flow when used from outside a request context.

We can create middlewares and http message handlers to have the exact behaviour we expect, specially after seen how it is done, however it leaves me feeling this "gap" where I see several packages officially supported by the AspNetCore team that don't play well with each other. I would never have expected to not be able to use the same typed httpclient on a service if that service is consumed by a controller and by a background service.

Would it be a problem if the header forwarder message handler simply did nothing if there was no available http context to get headers from?

[techgeek03]

Any conclusion on this? I'm with @raphaabreu on this. Throwing an exception if there is no available http context relay does not work in all cases. The best possible solution is to just remove the throwing of the exception and don't do anything.

In my case I'm using Refit and also resolving the typed clients in integration tests. We mainly want to use this package to pass diagnostics headers from envoy used for distributed tracing.

If the point of that exception is to detect misconstructions; you will know the moment the headers are not being propagated :)

[alefranz]

Hi @techgeek03,

The goal of detecting misconfiguration is to avoid the problem when the headers are not propagating as expected but it goes unnoticed. I think it is about striking the right balance between pushing cusers in the right direction versus allowing maximum flexibility and, at the time, it looked like there wasn't much interesting in supporting other scenarios.

Regarding your specific issues, you can use the HeaderPropagation middleware in the current state in integration tests. Are you including the middleware in your test server? Or what is the context in which you are experiencing this exception? Maybe I can help you find a workaround or having a concrete example could help make a case for this change.

However, the recommended solution for distributed tracing in ASP.NET Core does not involve this middleware (although I believe it still doesn't support having different header names, but I could be wrong).

[ththiem]

We recently ran into this issue that broke the entire call flows of our services as a result of this exception. There are numerous real world scenarios where service A makes http calls to service B for different purposes: It could be as a direct result of an incoming request to service A, it could also be processing messages out of an event queue, it could be a background task spawned by a request in any asynchronous scenario.

And the workaround of creating a duplicate http client just to not have the propagation is in direct violation of the recommended guidelines for proper httpclient lifetime management, where you are expected to reuse the same instance to take advantage of connection pooling to prevent socket exhaustion and reduce the delay in creation and teardown of the instances. So this really needs to be addressed sooner rather than later.

[Tratcher]

And the workaround of creating a duplicate http client just to not have the propagation is in direct violation of the recommended guidelines for proper httpclient lifetime management,

You're still expected to use a different HttpClient whenever the configuration is different. So long as you're re-using your clients and not throwing them away after each use you'll avoid the issues discussed in the guidelines.

[Rezakazemi890]

I’ve run into a practical scenario that I believe this issue affects, and I’d like to share it along with a potential improvement suggestion.

Scenario:
I have a TypedClient that is used in two different contexts within my application:

• During HTTP requests, where I want to propagate certain headers using the HeaderPropagation middleware.
• Within a BackgroundService or HostedService, where no active HTTP context exists.

As expected, when using the TypedClient from within an HTTP request, everything works fine. However, when it's called from the BackgroundService, I encounter the same InvalidOperationException mentioned in this issue because the HeaderPropagationValues have not been initialized.

Current workaround:
To handle this, we’ve had to create two separate implementations of the TypedClient:

• One configured with the HeaderPropagation handler (for use in HTTP requests)
• One without it (for background services)

This leads to duplicate code and maintenance overhead, which isn't ideal.

Proposal:
It would be helpful if the HeaderPropagation middleware offered an option within its configuration to control whether it should enforce the presence of an active HTTP context.
For example:

services.AddHeaderPropagation(options =>
{
    options.RequireActiveRequestContext = true; // default to true for backwards compatibility
});

When set to false, the middleware would simply skip propagation if no HTTP context is available, instead of throwing an exception. This would make it much easier to use the same TypedClient in both HTTP and non-HTTP contexts without duplicating the implementation.

Would love to hear what others think about this — I believe this could address several real-world scenarios without introducing breaking changes.

@alefranz @rynowak

[adamrodger]

I managed to work around this issue in a background service by adding a message handler to all my HttpClient instances (created via HttpClientFactory) which ensures that the header values are never null:

    internal class MyMessageHandler : DelegatingHandler
    {
        private readonly IServiceProvider services;

        public MyMessageHandler(IServiceProvider services)
        {
            this.services = services;
        }

        protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
        {
            // see: https://github.com/dotnet/aspnetcore/issues/12169 for details on why this is needed
            var values = this.services.GetRequiredService<HeaderPropagationValues>();
            values.Headers ??= ImmutableDictionary<string, StringValues>.Empty;

            return base.SendAsync(request, cancellationToken);
        }
    }

As far as I know that should still allow propagation to work when called from a controller context, but also not throw when used from a non-controller context like a hosted service doing background processing.

[Herzanet002]

@adamrodger You can also do it this way. First, register the collection as singleton. Then add the headers' propagation.

serviceCollection
            .AddSingleton<HeaderPropagationValues>(_ => new HeaderPropagationValues
                { Headers = ImmutableDictionary<string, StringValues>.Empty })
            .AddHeaderPropagation(options => { options.Headers.Add(HeadersConstants.HeaderKey); })

[Rezakazemi890]

@adamrodger You can also do it this way. First, register the collection as singleton. Then add the headers' propagation.

serviceCollection
            .AddSingleton<HeaderPropagationValues>(_ => new HeaderPropagationValues
                { Headers = ImmutableDictionary<string, StringValues>.Empty })
            .AddHeaderPropagation(options => { options.Headers.Add(HeadersConstants.HeaderKey); })

I've been using this method for a while now and it's working for me.

https://github.com/Rezakazemi890/PropagationMw