aspnetcore
aspnetcore copied to clipboard
dotnet
Multiple ASP.NET Core OAuth Correlation cookies
Is there an existing issue for this?
- [X] I have searched the existing issues
Describe the bug
We're using ASP.NET (v6) with Angular, using Cookie authentication to the back end, and proxying to an API (secured via OAuth). This is working the majority of time, but we are seeing occassions where the server is generating .AspNetCore.Correlation.oauth.$RANDOM cookies with a null expiry. Using Chrome, the header appears as:
set-cookie: .AspNetCore.Correlation.oauth.ativ9Y_r)bPnrqIOXsbWR5I.......; expires=Thu, 01 Jan 1970 00:00:00 GMT; Path=/
A correctly generated cookie during the OAuth cycle looks like:
set-cookie: .AspNetCore.Correlation.oauth.BSPC3Ydwq95MYo0t-T4MxWMrsIVN_q_zTwR-FhdgizA=N; expires=Fri, 15 Jul 2022 05:03:46 GMT; path=/signin-oidc; secure; samesite=none; httponly
(So secure, httponly are missing, and the expires option is incorrect)
The cookie is attached (sometimes) after the OAuth cycle, but the cookie is rejected and the Signin cycle starts again. After a failure, each subsequent request to /signin-oidc gets an additional set-cookie header (resulting in multiple cookies).
Nothing appears in the Asp.Net server logs, and clearing the cache resolves the issue.
We are running in a load-balanced environment (k8s+Docker) with multiple containers behind the same ingress URL.
Expected Behavior
The cookie is generated and passed around correctly.
Steps To Reproduce
Exact reproduction steps are unknown.
Exceptions (if any)
No response
.NET Version
5.0
Anything else?
No response
expires=Thu, 01 Jan 9170
😕
I'm assuming that's a typo for 1970? We should look at how we're generating these cookies. @Tratcher PTAL.
expires=Thu, 01 Jan 9170
😕
I'm assuming that's a typo for 1970? We should look at how we're generating these cookies. @Tratcher PTAL.
Yeah, sorry it was a typo. Should be 1970. Had to transpose from a screenshot.
expires=Thu, 01 Jan 1970 00:00:00 GMT; is generated to delete a cookie, which in this example always happens when visiting the /signin-oidc endpoint. Can you share a Fiddler trace, or at least the full Set-Cookie header?
That cookie shouldn't be included on future requests.
If secure missing that might be an issue with your proxy config. See https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/proxy-load-balancer?view=aspnetcore-6.0
Hi @danielbecroft. We have added the "Needs: Author Feedback" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.
Hey @Tratcher , thanks for the response. I'll try and capture a trace when we get the issue (it's very sporadic, and the symptoms that we're seeing now doesn't match the symptoms last week).
I've got a screenshot with the full set-cookie header on a previous occurrence, which is listed below. A new set-cookie header gets generated on each attempt to login, and all cookies get sent on the next request (they aren't getting cleared, based on your comments seems like something else).
I'll post back with more information if I can get a full stack trace and request history.
Yeah, those are all delete cookies, see the empty value and expires. It's weird to see more than one per request, it should only be deleting the one it used for that cycle. I've seen custom workaround code designed to mass delete these cookies, do you have anything like that? The deletes could fail if the attributes don't match the original, like httponly and secure. What does the initial set-cookie header look like when these are created at the start of the oauth flow?
What OAuth provider are you using? Can you show the setup code for that?
Hi @danielbecroft. We have added the "Needs: Author Feedback" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.
@Tratcher We're using an in-house OAuth identity server, but using the .AddOAuth() extension from Microsoft.AspNetCore.Authentication.OAuth, combined with cookie authentication, in a BFFE model . Our configuration code for the various providers is below.
services.AddAuthentication(options =>
{
options.DefaultScheme = "cookies";
options.DefaultChallengeScheme = "oauth";
})
.AddCookie("cookies", o =>
{
o.Cookie.Name = "__Host-applicationWeb";
o.ExpireTimeSpan = TimeSpan.FromMinutes(30);
o.Cookie.SameSite = SameSiteMode.Strict;
o.Events = new CookieAuthenticationEvents
{
OnValidatePrincipal = async context =>
{
// snipped - code to refresh `access_token` using stored `refresh_token`
}
};
})
.AddOAuth("oauth", o =>
{
o.AuthorizationEndpoint = new Uri(_settings.BaseUrl, _settings.AuthorizeEndpointPath).ToString();
o.TokenEndpoint = new Uri(_settings.BaseUrl, _settings.TokenEndpointPath).ToString();
o.ClientId = _settings.ClientId;
o.ClientSecret = _settings.ClientSecret;
o.CallbackPath = new PathString("/signin-oidc");
o.SaveTokens = true;
o.UserInformationEndpoint = new Uri(_settings.BaseUrl, "/api/v1/user").ToString();
o.Events.OnCreatingTicket = async context => { await CreateAuthTicket(context); };
o.Events.OnTicketReceived = context =>
{
/*
* In case Cookie is removed by browser automatically and user click on logout link. Then the ReturnUri
* will be keep as '/account/logout' instead of '/'. That is reason why we got the second login after
* the first success login. To fix it we will update ReturnUri to '/' to avoid the duplicate login issue.
*/
context.ReturnUri = context.ReturnUri.Contains("/account/logout") ? "/" : context.ReturnUri;
return Task.CompletedTask;
};
o.Events.OnRemoteFailure = context =>
{
_logger.LogError($"On remote failure: {context.Failure.Message}");
context.Response.Redirect("/");
context.HandleResponse();
return Task.CompletedTask;
};
});
Is there any way you can try this without loadbalancing and see if this is perhaps related to requests going to different containers?
Hi @danielbecroft. We have added the "Needs: Author Feedback" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.
This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment. If it is closed, feel free to comment when you are able to provide the additional information and we will re-investigate.
See our Issue Management Policies for more information.
Hi @HaoK , I've tried to enable load balancing on a duplicate environment, and it occasionally fails with the same result (but not always), so load balancing might be a potential problem. There's minimal output in the logs when this issue occurs, so it's hard to tell where the issue lies.
We are pushing out a change that ensures the /signin-oidc request bypasses the Angular service worker, but since we've had inconsistent reproductions both with and without this change, it's hard to know if this will resolve the issue.
Are there any events that I can hook into and add logging to try and get more information?
Our configuration code for the various providers is below.
Do you have multiple AddOAuth calls? In that case you must use unique CallbackPath values per provider. Otherwise multiple would be executed on a single request, and they'd all try to delete the cookie.
Hi @danielbecroft. We have added the "Needs: Author Feedback" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.
@Tratcher No, we only have a single .AddCookies() and .AddOAuth() call in our configuration.
https://docs.microsoft.com/en-us/aspnet/core/fundamentals/logging/?view=aspnetcore-6.0#configure-logging If you turn the existing logging up to Information or Debug that will help.
Reading back through this I realized it's not just the Secure attribute that's missing from the delete cookie, it's also missing samesite=none and httponly. That's suspicious. Delete used to leave those values off but that was fixed ~5 years ago. Can you confirm which framework version you're using? Your initial post said both 5 and 6.
Hi @danielbecroft. We have added the "Needs: Author Feedback" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.
This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment. If it is closed, feel free to comment when you are able to provide the additional information and we will re-investigate.
See our Issue Management Policies for more information.
Hi @Tratcher , sorry we're currently running .NET 5 in production (upgrading to .NET 6 in development at the moment). Once we've pushed v6 into production, we'll try and observe any changes.
Looking through other logs, we have noticed that our Web Application Firewall (WAF)_ is triggering on some AspNetCore.Correlation.oauth cookie value due to "SQL comment format structure", specifically where it has -- inside the string. This isn't causing a 403 error to be returned (which is what we'd expect), and also some of our users are using the application without going through the WAF so we're not sure if this is a red herring or not.
We'll increase the logging in production and see if there are any information that can be gleaned from it.
One interesting observation from the failing requests (only 2 today), the /signin-oidc call after the authentication is still sending the old __Host-supermateWeb cookie in the header. Login and timeout flows that don't send this request are not sending this cookie.
On working authentication flows, the Browser is still reporting that this cookie is valid (appears in the Cookie and Site data section with the old value, "session" expiry, everything). All these settings imply that it probably should be getting sent on every request to the app.example.com domain, but something is stopping it being applied (and I can't figure out what). There is no Set-Cookie response header to remove the header.
For example:
GET app.example.com
Cookie: __Host-myApplication=hunter2
HTTP 302 Found
Location: auth.example.com/oauth/v1/authorize
Set-Cookie: .AspNetCore.Correlation.oauth.RANDOM=N; expires=Mon, 22 Aug 2022 06:48:41 GMT; path=/signin-oidc; secure; samesite=none; httponly
(user authorizes)
POST auth.example.com/oauth/v1/authorize
HTTP 302 Found
Location: app.example.com/signin-oidc
GET app.example.com/signin-oidc
Cookie: .AspNetCore.Correlation.oauth.RANDOM=N; __Host-myApplication=hunter2
(Note the __Host-myApplication=hunter2 being sent on the /signin-oidc request).
My theory is that the presence of this cookie is triggering the authentication cycle again (as the cookie has expired), but I can't work out where this cookie gets removed during a working authentication flow. But I can't work out what's removing the cookie and/or stopping it from being sent on the /signin-oidc request?
What are the attributes on __Host-myApplication when it's created? E.g. if it has a Strict SameSite setting then it won't be included after a remote login flow. It would need to be set to Lax or None.
Looking through other logs, we have noticed that our Web Application Firewall (WAF)_ is triggering on some
AspNetCore.Correlation.oauthcookie value due to "SQL comment format structure", specifically where it has--inside the string. This isn't causing a 403 error to be returned (which is what we'd expect), and also some of our users are using the application without going through the WAF so we're not sure if this is a red herring or not.
That's unrelated as long as WAF isn't rejecting the requests. Their rules are unrealistic.
Hi @danielbecroft. We have added the "Needs: Author Feedback" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.
What are the attributes on __Host-myApplication when it's created? E.g. if it has a Strict SameSite setting then it won't be included after a remote login flow. It would need to be set to Lax or None.
The __Host cookie currently has SameSite=strict. The full Set-Cookie value is below:
Set-Cookie: __Host-supermateWeb=*****; path=/; secure; samesite=strict; httponly
You have raised an interesting point, though. The only place we can reliably replicate the issue is in production. In production, both the OAuth provider and application are subdomains of the same registered domain (api.example.com and app.example.com). All our other environments have the application and authentication provider on different registered domains (review-app-123.example.net and api-dev.example.com).
Reading up on SameSite=Strict v SameSite=Lax options, this could be at least part of the problem, if not the entire problem. Even though both URLs are on different subdomains, the handling of SameSite only looks at the registered domain (example.com). This means that if they are both in the same registered domain, then the HTTP 302 Found response after authenticating, is no longer consiered a "cross-site" request, and the cookie gets sent on the /signin-oidc request.
I've proven this concept by changing our testing environments to use the SameSite=Lax setting, and we know reliably trigger the issue. The ExpireTimeSpan runs out, the authentication flow is triggered, and the __Host cookie is send on the /signin-oidc request.
This then all seems to be related to our use of ExpireTimeSpan on the cookie configuration:
o.ExpireTimeSpan = TimeSpan.FromMinutes(30);
(Why this is only happening for some users, on some browsers, is still unclear)
We used ExpireTimeSpan instead of CookieOptions.MaxAge, as we need the user to login after 30 minutes of inactivity, and we wanted the cookie to be removed when the browser window closes. The caveat to using that option, though, is that we now end up with a cookie who's authentication ticket has expired, but never gets deleted by the browser, and the user is never able to renew without deleting the cookie.
It seems that we can't have both, and unfortunately the 30-minute expiry is a regulatory requirement, so we'll switch to using CookieOptions.MaxAge instead of ExpireTimeSpan, and see if that resolves the issue.
Update: CookieOptions.MaxAge doesn't have the sliding facility, so we're locked to a 30-minute window before forcing a login, even if there has been activity.
What issue are you having with expired tickets? They should be ignored and trigger a new auth flow, replacing the cookie.
Hi @Tratcher , I think the core of the problem is that we are expiring the authentication ticket portion of the cookie, but not removing the cookie itself. This leaves the (now invalid) cookie in the browser and sends it on the second GET /signin-oidc request. We're solving this by setting CookieOptions.MaxAge and CookieAuthenticationOptions.ExpireTimeSpan like so:
o.ExpireTimeSpan = TimeSpan.FromMinutes(30);
o.SlidingExpiration = true;
o.Cookie.MaxAge = o.ExpireTimeSpan;
o.Cookie.SameSite = SameSiteMode.Strict;
This ensures the max-age property on the cookie is set to the same time as the authentication ticket expiry, so when it expires, the cookie doesn't get sent, and we get a new cookie back.
The more I've looked into this issue, the more I think it's actually linked to some incorrect code we have implemented checking the context.User.Identity.IsAuthenticated property during the pipeline. We're going to reexamine that after we've fixed the login loop problem.
I'll report back if this actually fixes the problem in production and the issue can be closed.
A cookie containing an expired auth ticket should be harmless, context.User will be populated with an anonymous user.
Thanks @Tratcher , it is getting populated with an anonymous user, but some of our code checks this, and issues a challenge immediately. It's this logic that I now think is actually incorrect, so we'll look at fixing that later.
Our change to use MaxAge as well as ExpireTimeSpan has resolved the issue. The MaxAge stops the cookie getting sent on the subsequent /signin-oidc request, and the loop and multiple cookies no longer occurs.
I'm going to close this issue now, as we have been able to resolve it. Thanks @Tratcher and @HaoK for your assistance.