HTTP 400 Bad Request errors

[kcgthb]

Hi!

We have a number of users periodically reporting HTTP 400 errors when accessing the On Demand dashboard:

Bad Request
Your browser sent a request that this server could not understand. Size of a request header field exceeds server limit.

In each case, it seems that deleting the _dashboard_session cookie fixes the issue and allows the user to load the page correctly.

We're using OnDemand v1.6.22 (dashboard v1.35.3). Does that sound familiar, and is this something that we could avoid somehow?

┆Issue is synchronized with this Asana task by Unito

[johrstrom]

Hey! We've seen this before here at OSC too. Thanks for bringing it up! I usually tell folks to clear all cookies for that site.

Do you happen to use OIDC? We enabled an oidc setting OIDCStateMaxNumberOfCookies so that the oldest oidc cookies get deleted. I'm not sure if that fixed this issue, but surely didn't hurt.

So, it sounds familiar, but I'm not sure how to avoid it. Which is to say, we should probably look into why it's happening and document to avoid it (and/or fix it if we're causing it).

[ericfranz]

We are hearing about this more often. We should do a review of OnDemand at OSC and enumerate here all the possible cookies, with their Domain, Path, and other attributes, to assess what we can do to help us and other sites avoid this in the future, or at least more quickly diagnose the problem.

1. For example, if it is just properly configuring OIDC with OIDCStateMaxNumberOfCookies, that could be added to the or at least documented.
2. For app created cookies we could also avoid apps creating their own cookies by having one user settings or state cookie, avoid using cookies altogether, or fix the Path to cookies so that a request to the dashboard doesn't send a cookie for active jobs, and vise versa - though I thought we had fixed that problem several years ago.

[treydock]

For the OIDC aspect, we could add a sane default with the Dex support as OIDC configs become directly configured with the changes to support Dex.

[kcgthb]

Sorry for the delay answering! We actually don't use OIDC, we use Shibboleth for authentication. Is there a similar setting?

[ericfranz]

We have not yet been able to produce the problem at OSC, though there was a time when we had the problem with many mod_auth_openidc cookies being generated for OnDemand. Unfortunately I do not recall the configuration change that addressed that.

In your cookie list you have a number of cookies that do not look familiar: the four AMCV_ ones and pgn_user2 for example.

The only thing that uses the _dashboard_session cookie in OnDemand, as far as I can tell (unless I missed something), is the Rails flash object for passing temporary messages between page redirects. So you submit a web form, if it succeeds (or fails) it might redirect you to another page - and it adds the message to display when you request that page to the cookie. It is temporary - so it is removed as soon as the page displaying the message renders.

My current theory is that unrelated cookies build up from other sources that do not have the proper path scope - and thus the cookie header in each request to OnDemand keeps growing. At a certain point when the _dashboard_session has a larger flash message it ends up putting the cookie length over the limit, causing the error.