whatwg
Why does Fetch specify a max redirect count of 20?
Why does Fetch specify a max redirect count of 20? See https://fetch.spec.whatwg.org/#http-redirect-fetch (step 5)
It seems Chrome and Firefox have implemented this. I wanted to align Firefox but found out that CFNetwork currently uses a max redirect count of 16 internally which is not configurable. This is preventing us (and any CFNetwork client) to align with the Fetch specification.
Would people consider lowering the max redirect count in the Fetch specification given this (new?) information? The alternative for us would be to ask CFNetwork to make it configurable but this would likely take time.
@cdumez Asking folks internal to see if there's any Ancient Wisdom behind this limit, but from source archaeology for Chrome, our limit of 20 was chosen to match Gecko/Firefox at the time we first open-sourced
See https://chromium.googlesource.com/chromium/src/+blame/5cbb127e6b67a646b98a460faf89740f29b91403/net/url_request/url_request.cc - which traces its provenance of kMaxRedirects to Gecko.
Given that Safari is limiting to 16, that seems compelling enough to suggest there is limited risk, and interoperability is always good. Barring any concerns being shared, I think we'd be open to aligning with Safari.
FYI, from HTTP 1.1 RFC:
https://tools.ietf.org/html/rfc7231#section-6.4
A client SHOULD detect and intervene in cyclical redirections (i.e., "infinite" redirection loops).
Note: An earlier version of this specification recommended a maximum of five redirections ([RFC2068], Section 10.3). Content developers need to be aware that some clients might implement such a fixed limitation.
cc @annevk
If you can agree on a common limit between browsers, we can refine this in the next HTTP; tell us on httpwg/http11bis if so.
This changed in gecko 15 years ago:
https://bugzilla.mozilla.org/show_bug.cgi?id=173147
We went from a max of 10 to 20 because "it seems that nytimes.com has been known to string together more than 10 redirects". Its possible nytimes.com has changed since then.
The limit of 10 was introduced here based on lynx's behavior:
https://bugzilla.mozilla.org/show_bug.cgi?id=83471
Prior to that it appears we just looped forever.
One thing that may present a challenge for us lowering - we consider certain types of internally-generated rewrites (extensions, HSTS) as redirects for purposes of redirect counting. I believe, perhaps incorrectly, that Firefox does something similar with HSTS rewrites.
@cdumez Do you know what CFNetwork does in that space? I don't know if CFNetwork handles HSTS or whether that's added by the Safari layering atop, and I don't know whether or not that's counted as a redirect for purposes of limits.
Yea, we do internal redirects for all kinds of things and they appear to be included in our limit checking:
http://searchfox.org/mozilla-central/source/netwerk/protocol/http/HttpBaseChannel.cpp#3381
@wanderview Does Firefox have metrics on redirect counts, to assess what the impact would be?
We don't have general redirect counts, but Chrome 61 added one for top-level navigations ( PageLoad.Navigation.RedirectChainLength in https://cs.chromium.org/chromium/src/chrome/browser/page_load_metrics/observers/core_page_load_metrics_observer.cc?rcl=413e72f9569c5b502335883143d9b96f6cd1474c&l=264 ), but that'll take a while before we get representative samples, and wouldn't count sub-resources.
I suspect given the HSTS+Extensions implications, we'd probably want to measure to see what our breakage rate would be. Even from those preliminary results (from Beta users), we do see that there's a non-zero (roughly .003% from the current dataset) of users with >16 redirects in the top-level navigation.
I don't see any telemetry probes for the keyword "redirect" on https://telemetry.mozilla.org.
So, I've filed https://bugs.chromium.org/p/chromium/issues/detail?id=755117 to measure this, given our differences re: extensions, to assess the compat risk. This will help gather data about our compat risk and see if we anticipate any risk with changing to 16.
It may also be worthwhile for Chrome and Firefox folks to discuss the implications of extensions/HSTS counting in that limit. I think from a Chrome side, it keeps the code much simpler "as implemented" (hence why we did it), but it potentially represents an observable compat difference for sites, since a page load that might work in Safari (with no extensions) might fail in Chrome (with extensions). I'm not sure if we care about predictability to that degree of specificity, hence I didn't directly open a bug, but if folks say "Huh, good point", then we should track that somewhere :)
@sleevi it does seem important to track, especially given HSTS and Upgrade-Insecure-Request. @mikewest at some point asked the standard to change its model to have these kind of "internal redirects", but didn't follow up on the questions I asked: https://github.com/whatwg/fetch/issues/324. What the specification does seems simpler to me, but I might be missing something.
@annevk Exposing internal redirects significantly complicates the API surface. I realize that's an "implementor concern", but one pragmatically chosen to maintain a well-defined separation of layering between components. We've evaluated tracking this in several other contexts/feature bugs, and somewhat consistently arrived at the conclusion that the complexity and cognitive overhead of the API doesn't match how our consumers (in Chrome) use it.
From a server viewpoint, as @mnot mentioned, servers should be prepared for clients that accept less, and I also totally appreciate an argument that "extensions can change how Fetch is observed". So I don't think it's terribly important that we track, but I also totally appreciate that I may be missing some considerations.
Anyways, the metrics bug should get us some data points in a few releases (looking at the calendar, looks like Chrome 62). Hopefully Firefox will considering adding similar telemetry and we can see if reducing to 16 is viable :)
I don't care much for extensions, but I do care about HSTS and Upgrade-Insecure-Requests which are not extensions.
We use internal redirects to do things like "reset the network request" after a service worker decides not to intercept, etc. Probably a fair number of internal redirects in firefox.
Naive question for implementers, is it possible for internal redirects to not increase the redirect-count, at least as compared to the limit? (Or, equivalently, for internal redirects to increase the limit, for that fetch at least.)
(My perspective is just that it sure would be nice if everyone was observably the same :) )
Naive question for implementers, is it possible for internal redirects to not increase the redirect-count, at least as compared to the limit?
Yea, we could probably do that. We definitely have all the information in the same place. We just have to check the flag.
@domenic No, because for Chrome, we intentionally do not make a distinction between internal and external redirects, because as an API concept, they do not align with the design or goals of the network stack (and it's use across multiple Google products).
I can understand the principle of wanting things observably the same, although I suppose I disagree with it as a general rule (I don't think every API and behaviour should be overly specified, as otherwise that adds unreasonable overhead and stop energy towards evolving and experimenting). Given that the specs explicitly call it out as an implementation specific behaviour, I think it may be overspecified in Fetch to include such a limit, but I don't have the energy to argue for that separation, just that we don't further constrain things without good justification :)
Well, it sounds like we maybe able to get this to be an everyone-but-Chrome standard, at least...
@domenic That reads as unnecessarily snarky and hostile, but I could be misreading your tone.
In the interest of finding a solution that works for everyone, rather than change for change sake, could you perhaps elaborate why all redirects shouldn't be counted as just that - redirects? Treating HSTS towards the limit does not seem wrong or incorrect, perhaps you could share if you feel otherwise and why?
Maybe I am conflating Firefox's internal redirects and Chrome's, but it sounded like internal redirects were very implementation-specific, and not limited to well-known cases that everyone would interoperate on.
Reading up, it does seem you are consistently saying "HSTS+extensions". So apart from extensions, is it indeed only HSTS redirects? If so then we may indeed be in a slightly better position, although it sounds like Safari and Chrome's network stacks are somewhat opposed and neither are easy to change...
@domenic I don't think they're opposed. Chrome's limit of 20 is much easier to change than Safari's, provided we get metrics on risk. It's unclear the overhead of changing HSTS to be counted as redirects for purposes of Safari - @cdumez hasn't weighed in on that, and I don't want to assume. So it's possible that if we don't change to a segmented internal/external redirect, and given that two implementations don't (presently) make that distinction, we could potentially align on a 16-redirect limit and call it done. That's why I did not understand the advocacy for an internal/external split - it's something we could potentially do without.
That said, I think it's also useful to understand what impact, if any, that internal/external split has on the Web Platform. To the extent the Fetch spec represents "API docs", I'm totally on board with documenting API contracts, but I don't think we should overly specify those API contracts in a way that prevents us from making changes, integrations, or improvements. That's why I think over-specifying redirects would be undesirable, and it doesn't sound like we have a compelling reason to make that split other than "We should document that" - that is, it's unclear how or why that should cause different behaviour.
@annevk identified one other - which is Upgrade-Insecure-Requests. But it's unclear to me what, if any, benefit is derived from having HSTS or U-I-R not treated as redirects. We could equally specify that the application of these counts as a redirect for purposes of Fetch's limit, and be done with it.
So the concrete question is:
- Would implementations have issue with treating HSTS and U-I-R (e.g. specified bits, as opposed to internal implementation details) as counting as redirects towards the limit? Would that present challenges for implementations commiserate with the challenges of splitting internal/external?
Upgrade-Insecure-Requests happen at WebKit-level and we could probably easily count them towards a WebKit-level max redirect count if we wanted to. Given that CFNetwork would go to up to 16 HTTP redirects, WebKit can artificially limit further by taking into account U-I-R.
HSTS is another issue though. I believe this happens below WebKit, at CFNetwork level. I'll ask but I am not sure WebKit is aware of HSTS currently.
Well, for Firefox there's also the service worker not handling the request per @wanderview. And I wouldn't count on all that being exhaustive.
Also, I don't understand how these internal redirects in Chrome are not a little bit special. It seems CORS would break if they're not.
@annevk Our network stack takes a different approach than Firefox, and is more similar to the CFNetwork+WebKit split or WinHTTP+Edge split - two separate layers. HSTS is implemented by our //net layer, and is thus opaque in handling and processing to the Chrome (and, from the POV of Chrome, Fetch) layer. This is where the opposition to internal/external comes from - it is, from an API perspective, exposing an implementation detail of our networking stack, and one whose ontology differs based on different consumers (e.g. whether a given redirect is injected from a higher layer - e.g. U-I-R or extensions - or comes from a lower layer - like HSTS - can result in different categorizations based on different //net consumers' needs). This is further amplified by future steps Chrome is taking to move its networking into a dedicated sandboxed process (and potentially consumed by non-Chrome applications)
In evaluating whether or not to make this split (which affects a variety of features that have redirect-specific behaviours, such as our omnibox search logic), we've consistently decided that treating them holistically is more desirable and more understandable for the various //net consumers. Thus, we've allowed a small CORS layering violation to be introduced into //net, rather than force all //net consumers to be aware of the ontology (or to support extending the ontology, as necessary for some of these consumers). That logic is in https://cs.chromium.org/chromium/src/net/url_request/url_request_redirect_job.cc?rcl=49b1594550778be788e86953fe6d17401e96b5f8&l=100 (while that code is shared by some, but not all, of internally generated redirects), as detailed in https://chromium.googlesource.com/chromium/src/+/4e0be1f338c3bb9bb3c347cdb4c5e3b7944c5c67
Not considering the the difference in what counts as a redirect in various engines for a second, and given @mnot 's comment from earlier. I'll mention that if the limit of 20 were to be spec'd in an HTTP RFC, it would probably be easier to get CFNetwork updated. The limit of 20 in the Fetch spec is unfortunately not a very strong argument to get such change done in our lower networking stack level.
Otherwise, I think Chrome gathering data on how safe it would be for them to lower their limit to 16 sounds like a good idea. Just note when adding such logging that we'd want to know if users experience over 16 redirect but the load actually succeeds (i.e. total redirect count was not over 20).
@cdumez Yeah, my plan was to count the final redirect count for successful loads and measure that - which would give an indication for loads in the 17, 18, 19, and 20 range (which would stop working), while naturally wouldn't report for redirect counts >20 (which would count as failed). Did I correctly understand your remarks?
FYI, I am in the process of moving our limit on "internal" redirects to be separate from the "real" redirect count. This would make it easier to align our redirect limit with other browsers in the future if that is decided.
My work is in:
https://bugzilla.mozilla.org/show_bug.cgi?id=1405739