pshtt
pshtt copied to clipboard
cisagov
Factor in redirects when calculating HSTS
I mentioned some of the set up to this problem in the TTS #https-partner-support Slack channel, but the long and the short of the issue is that in https://github.com/dhs-ncats/pshtt/blob/00ff246f40acbea185d478d838c7fcd6652b9aa8/pshtt/pshtt.py#L80 the check is being done on "not the final https URL" as best I can tell (possibly it's actually randomly doing this...).
So for instance, as I'm looking at lc.llnl.gov, the redirects are:
- http://lc.llnl.gov/
- https://lc.llnl.gov/
- https://lc.llnl.gov/nclogin.submit?LOGIN
And only the last hop (final URL) has the HSTS bits set up. This is the case for both lc.llnl.gov and lc-idm.llnl.gov, but for mylc.llnl.gov (which pshtt is currently saying is good for HSTS) the only difference I can find is in the middle hop:
- http://mylc.llnl.gov/
- https://lc.llnl.gov/lorenz
- https://lc.llnl.gov/nclogin.submit?LOGIN
Although, it also is not showing HSTS on the middle hop.
I'm still continuing to dig into this, but thought I would get the ticket opened while I work (since I still can't contribute directly yet).
FWIW, this is currently understood behavior, and was intended when it was shipped. So this is more of a feature request than a bug report.
That said, this request has come in a fair amount, especially since SSL Labs started factoring in redirects when calculating HSTS support. I think it's a reasonable request, as long as it's not hugely complicating logic.
@h-m-f-t What do you think?
Good to know. I guess then the question becomes (if this isn't a feature to be added) what is the preferred way to resolve this?
Would it simply be to have every HTTPS url in the chain all be HSTS? I suppose that makes a certain amount of sense, just isn't what I was expecting. Maybe we could do something like add a warning or something to denote it as a concern?
So far, we've pushed folks to have the HSTS header on the root path (/), which should get HSTS marked as a True in pshtt today.
@IanLee1521 Thanks for filing this issue (and others). I promise we're not ignoring #72.
I'm sympathetic to your request, and it's definitely one NCATS hears with some frequency. My understanding is that once seen by a browser, HSTS, whether seen initially on example.gov or example.gov/program will be protective across the entire base domain.
From the vantage point of a centralized scanning service, one where we have a nominal list of .gov domains and the hosts using it, not accounting for redirects makes scanning and scoring easier. It also follows a hard-line reading of M-15-13, where HSTS use is mandated on all HTTPS-using sites. Accounting for HSTS solely at the final redirected destination would technically be ignoring non-compliant behavior at all the interim HTTPS redirects.
I don't want compliance for compliance' sake, or maximal-compliance for a little security benefit— and the current approach does have a bit of that flavor. Happy to hear more discussion, but I think I'm supportive of making this change, especially given that SSL Labs does this.
Understood. Thanks for the explanation.
FWIW, partially this was me being in the middle of our internal folks making the changes and my running the tool. And the case of SSL Labs vs pshtt (vs curl as I understand it) was what seemed to be causing the confusion.
I've gone ahead and pointed our internal folks here to this thread for the context, and asked that they make the change to the root endpoint for https://lc.llnl.gov/ so that we can get this cleaned up (and both tools will agree).
That being said, I'm already starting to dream up how we might present a warning to a user running this to try to better communicate this issue, and what the fix would be. It's one of a number of improvements I would love to propose, provided I can ever get there... ;)
I am Balaji from Citrix networking team and it is an load balancing product This issue is seen for one of our customer who has load balanced his application using our product Is there any ETA for this issue
@bpanchan There's no ETA I'm aware of for this issue. We welcome open source contributions.
We already capture the final response from redirects in ultimate_req. Can we just use that response to see if the site is still the same and the headers include HSTS if the root response didn't include HSTS, or would we need to actively check every response if there are multiple redirects? If it's just the final response, I can go ahead and make the change.
We just got a request about this same issue for an appliance that has hard-coded HSTS, but it's not on the first response that immediately redirects, it's on the next response where the redirect lands.
FWIW, we're (Gov Canada) looking at this issue and treating all endpoints as separate domains -- just because a root also has a www doesn't mean that the root should inherit the www results, and are grappling with that as the domain scanner always looks at each endpoint (all 4) every time it does a scan. HSTS for www shouldn't be inherited for root, or vice versa -- definitely not from an external domain if it redirects off the main root elsewhere entirely.
The issue referenced above has more.
Direction we're giving is to secure each endpoint prior to redirecting, including HSTS.
@tallardyce, that makes sense since HSTS only applies to each specific host name individually. pshtt looks at HSTS for each endpoint individually, but then the pshtt results only reports the HSTS status of what it considers the canonical endpoint.
For the additional HSTS checks on redirects, they should only check for HSTS if the redirects are on the same endpoint and ignore them otherwise. These additional checks on the redirects on the same endpoint will help many sites. We see this in many cases where there is a Web Application Firewall (WAF) that automatically redirects from the root of the website to a consent page and/or a login page further in at a longer URL, but that the WAF doesn't include the HSTS header while the actual page that it redirects to and is logically on the same endpoint includes the HSTS header.
See https://github.com/cisagov/pshtt/commit/4861dd35eba1ecf025e8f8023a1baab157137fed for suggested changes to look for HSTS on all redirects. Is that what we should do?
How about for Server Name Indication (SNI)? We have a couple sites that are using the same host, but are using SNI so that we can use the same server for multiple sites. We implemented HSTS using an F5 LTM http profile with HSTS turned on. We noticed that both pshtt and SSLabs both report that HSTS is not implemented still.
Any thoughts? And thank you in advance!
@jdawg06, pshtt uses requests to perform the web requests and get the HSTS header. requests natively supports SNI automatically so pshtt should be connecting to the individual sites already and if there are HSTS headers, it should already see them. Can you provide some specific sample sites that are having this issue?
@jdawg06,
pshttusesrequeststo perform the web requests and get the HSTS header.requestsnatively supports SNI automatically sopshttshould be connecting to the individual sites already and if there are HSTS headers, it should already see them. Can you provide some specific sample sites that are having this issue?
account-fqt.etc.uspto.gov
I ran it against the Qualys SSL Test and it didn't pass HSTS. However, when i use Chrome DevTools and go under Network > Headers, its there. Configured with the max-age, and "includeSubDomains".
@jdawg06, the problem is not SNI. The issue is that the initial response for https://account-fqt.etc.uspto.gov does not include an HSTS header, but instead uses Javascript to set a cookie and reload the page. Once the cookie is set, then there are other responses and redirects that include the HSTS header, but the scanners (both pshtt and Qualys SSL Test) don't interpret the Javascript to eventually find the HSTS header. It would be better to add the HSTS header on the initial response, or at least set the cookie and refresh/redirect the page using HTTP headers that the scanners can follow (if they allow cookies and follow redirects).