[tracking] typed headers

[yoshuawuyts]

This is a tracking issue for typed headers. I looked over the HTTP headers page and categorized which headers would need to be ported.

There are probably some nuances in how we want to implement them, but this provides an overview of how far along we are in constructing typed constructors for all header types.

edit: I realize now that this might read a bit dated. I drafted this text back in December but forgot to post it. Some things may seem a bit outdated or different; the most important part is that we can track whether we can construct headers .

Authentication (auth)

• [x] WWW-Authenticate (auth::Authenticate)
• [x] Authorization (auth::Authorization)
• [x] Proxy-Authenticate (auth::ProxyAuthenticate)
• [x] Proxy-Authorization (auth::ProxyAuthorization)

Caching (cache)

• [x] Age (cache::Age)
• [x] Cache-Control (cache::Control)
• [x] Clear-Site-Data (cache::ClearSite)
• [x] Expires (cache::Expires)
• [x] ~~Pragma (cache::Pragma)~~ (HTTP/1.0 only)
• [x] ~~Warning (cache::Warning)~~ (soon to be deprecated)

Client Hints (client_hints)

• [x] ~~Accept-CH (ch::Accept)~~ (experimental, implementation postponed)
• [x] ~~Accept-CH-Lifetime (ch::Lifetime)~~ (experimental, implementation postponed)
• [x] ~~Early-Data (ch::EarlyData)~~ (experimental, implementation postponed)
• [x] ~~Content-DPR (ch::ContentDpr)~~ (experimental, implementation postponed)
• [x] ~~DPR (ch::Dpr)~~ (experimental, implementation postponed)
• [x] ~~Device-Memory (ch::DeviceMemory)~~ (experimental, implementation postponed)
• [x] ~~Save-Data (ch::SaveData)~~ (experimental, implementation postponed)
• [x] ~~Viewport-Width (ch::ViewportWidth)~~ (experimental, implementation postponed)
• [x] ~~Width (ch::Width)~~ (experimental, implementation postponed)

Conditionals (conditionals)

• [x] Last-Modified (conditionals::LastModified)
• [x] Etag (conditionals::Etag)
• [x] If-Match (conditionals::IfMatch)
• [x] If-Modified-Since (conditionals::IfModifiedSince)
• [x] If-Unmodified-Since (conditionals::IfUnmodifiedSince)
• [x] Vary (conditionals::Vary)

Content Negotiation (content)

• [x] Accept (content::Accept)
• [x] ~~Accept-Charset (content::Charset)~~ (no longer supported by any browser)
• [x] Accept-Encoding (content::Encoding)
• [ ] Accept-Language (content::Language)

Controls (controls)

• [x] Expect (controls::Expect)

Cookies

I think this one is special, and we should re-export the cookie crate, and just expose get / set cookies as methods on the Request and Response types. Let's just treat them as built-ins to evade the name collision.

CORS (cors)

• [x] Access-Control-Allow-Origin (cors::AllowOrigin)
• [x] Access-Control-Allow-Credentials (cors::AllowCredentials)
• [x] Access-Control-Allow-Headers (cors::AllowHeaders)
• [x] Access-Control-Allow-Methods (cors::AllowMethods)
• [x] Access-Control-Expose-Headers (cors::ExposeHeaders)
• [x] Access-Control-Max-Age (cors::MaxAge)
• [x] Access-Control-Request-Headers (cors::RequestHeaders)
• [x] Access-Control-Request-Method (cors::RequestMethod)
• [x] Access-Control-Origin (cors::Origin)
• [x] Access-Control-Timing-Allow-Origin (cors::TimingAllowOrigin)

Do Not Track (privacy)

• [ ] DNT (privacy::DoNotTrack)
• [ ] Tk (privacy::TrackingStatus)

Content Information (content)

• [ ] Content-Disposition (downloads::Disposition)
• [x] Content-Length (content::Length)
• [x] Content-Type (content::Type)
• [x] Content-Encoding (content::Encoding)
• [ ] Content-Language (content::Language)
• [x] Content-Location (content::Location)

Proxies (proxies)

• [x] Forwarded (proxies::Forwarded)
• [ ] Via (proxies::Via)

Response (response)

• [x] Allow (response::Allow)

Range (content)

• [ ] Accept-Ranges (range::Accept)
• [ ] Range (range::Range)
• [ ] If-Range (range::If)
• [ ] Content-Range (range::Content)

Security (security)

• [ ] Cross-Origin-Opener-Policy (security::Coop)
• [ ] Cross-Origin-Resource-Policy (security::Cors)
• [ ] Content-Security-Policy (security::Csp)
• [ ] Content-Security-Policy-Report-Only (security::Cspro)
• [ ] Expect-CT (security::Cspro)
• [ ] Feature-Policy (security::FeaturePolicy)
• [x] ~~Public-Key-Pins (security::HPKP)~~ (deprecated)

Other

• [ ] Alt-Svc
• [x] Date
• [x] ~~Large-Allocation~~ (unspecced; firefox only)
• [ ] Link
• [x] Referer
• [x] Retry-After
• [x] SourceMap
• [ ] Upgrade
• [ ] User-Agent
• [ ] Keep-Alive

[Licenser]

A good part of them are implemented in #107

[yoshuawuyts]

Example for a typed expires constructor:

use http_types::cache::Expires;
use std::time::{SystemTime, Duration};

let when = SystemTime::now() + Duration::from_secs(60 * 60);
let expires = Expires::new(when);
res.insert_header(expires.name(), expires.value());

[ririsoft]

Hello @yoshuawuyts ,

I have implemented a basic HTTP Range support (no multipart support) for a private project of mine using http-rs/tide. I have implemented a serve_content function, taking inspiration from golang net/http, for serving bodies implementing async Seek.

I would be happy to help implementing this back to http-types, async-h1 and tide. I believe things start with http-types but I am not sure about the API your are expecting "typed headers" to follow. Should I start with your example above and submit a PR for review ?

[yoshuawuyts]

@ririsoft hey! -- that's a good question. I'm not sure either what such an API should look like, but I'm sure we can figure it out. Opening a PR with a usage example would indeed be a great starting point.

[ririsoft]

Hello !

Opening a PR with a usage example would indeed be a great starting point.

I will submit one shortly for further brainstorming, addressing the HTTP range headers parsing and validation use case.

[ririsoft]

After my implementation of range request headers (PR #180) I am not convinced that a typed header will bring much value over the existing http_types::headers::ToHeaderValues and std::convert::TryFrom<&HeaderValue>.

The only added value I see is the automation of header name handling when fetching header values. For instance we could have the following (simplified version):

Gist: https://gist.github.com/rust-play/07ab2a72218d80531af7225dc0579c17

use std::collections::HashMap;
use std::str::FromStr;
use std::string::ToString;

// Here FromStr should be TryFrom<&HeaderValue>
// and ToString should be ToHeaderValues.
trait TypedHeader: FromStr + ToString {
    const HEADER_NAME: &'static str;
}

struct Headers {
    vals: HashMap<String, String>,
}

impl Headers {
    fn get<T: TypedHeader>(&self) -> Option<Result<T, <T as FromStr>::Err>> {
        self.vals.get(T::HEADER_NAME).map(|v| T::from_str(&v))
    }

    fn set<T: TypedHeader>(&mut self, val: T) {
        self.vals
            .insert(T::HEADER_NAME.to_string(), val.to_string());
    }
}

#[derive(Debug, Eq, PartialEq, Clone)]
struct Range {
    start: u64,
    end: u64,
}

impl ToString for Range {
    fn to_string(&self) -> String {
        format!("bytes={}-{}", self.start, self.end)
    }
}

impl FromStr for Range {
    type Err = &'static str;

    fn from_str(s: &str) -> Result<Self, Self::Err> {
        let err = "invalid range header";
        let mut bounds = s.trim_start_matches("bytes=").split('-');
        let start = bounds
            .next()
            .ok_or(err)
            .and_then(|v| u64::from_str(v).map_err(|_| err))?;
        let end = bounds
            .next()
            .ok_or(err)
            .and_then(|v| u64::from_str(v).map_err(|_| err))?;
        Ok(Range { start, end })
    }
}

impl TypedHeader for Range {
    const HEADER_NAME: &'static str = "Range";
}

fn main() {
    let mut h = Headers {
        vals: HashMap::new(),
    };
    let r = Range {
        start: 10,
        end: 100,
    };
    h.set(r.clone());
    let other = h.get::<Range>();
    assert_eq!(Some(Ok(r)), other);
}

We would need to add some Headers::set_values(&mut self, name: impl Into<HeaderName>, val: imp ToHeaderValues) and get_values equivalent, as well as append methods.

Actually the true additional value of the above implementation is the possibility to get the first header value instead of having to get all headers value and map option to HeaderValues::last().

What do you think ?

[Fishrock123]

I think we may be able to think about this better.

After thinking about this since yesterday, I was thinking about my wip port of negotiator - i.e. Accept-* headers, and realized we may want to specify "typed headers" for both Request and Response the same way:

This is derived from the neat api of accepts, a JS lib which builds on negotiator:

  var accept = accepts(req)
 
  // the order of this list is significant; should be server preferred order
  switch (accept.type(['json', 'html'])) {
    case 'json':
      res.setHeader('Content-Type', 'application/json')
      res.write('{"hello":"world!"}')
      break

i.e. You could specify a Range or an Accept like a request would, but also be able to match from it:

req.set_header(AcceptEncoding::new(["gzip", "deflate", "identity"]));

let accept: Option<AcceptEncoding> = res.get_header(AcceptEncoding::new(["gzip", "deflate", "identity"]));
match accept?.get_preferred_match() {
  "gzip" => ...
   ...
};

in this case, Headers::get()/set() would have to end up being something like:

impl Headers {
    fn get<T>(&self, mut header: T) -> Option<T> where T: TypedHeader {
        self.vals.get(T::HEADER_NAME).map(|v| header.populate(&v))
    }

    fn set<T>(&mut self, header: T) where T: TypedHeader {
        self.vals
            .insert(T::HEADER_NAME.to_string(), header.to_string());
    }
}

[ririsoft]

Hi @Fishrock123,

I did not know about negotiator, this is interesting. Thanks for sharing !

It seems we are ending with 2 close APIs with some subtle differences that suite best one case or another. The set method are the same (which is excellent), so let's focus on the get one.

// V1
impl Headers {
    fn get<T>(&self, mut header: T) -> Option<T> where T: TypedHeader {
        self.vals.get(T::HEADER_NAME).map(|v| header.populate(&v))
    }
}

// V2
impl Headers {
fn get<T>(&self) -> Option<Result<T, <T as FromStr>::Err>>  where T: TypedHeader {
       self.vals.get(T::HEADER_NAME).map(|v| T::from_str(&v))
   }
}

I need to mature more on the differences of the two design, please excuse me if I misunderstood some points, but here is what comes to my mind:

When fetching a header value from string we might want to do the following steps:

1. Know whether it is missing or not.
2. Validate its format against related HTTP RFC.
3. Do additional computation specific to the header type.

My concern with V1 design is that it merges all steps above. If I get a None I do not know if this is the result of step 1, 2 or 3. In the case of AcceptEncoding this might not be an issue, but in the case of Range this is because the response flow and status code are different at each of the 3 steps above.

V1 could return a Option<Result<T>> like V2 to address step 2, but it still does not make possible to distinguish step 1 from step 3, unless I missed something.

[Fishrock123]

// V2
impl Headers {
fn get<T>(&self) -> Option<Result<T, <T as FromStr>::Err>>  where T: TypedHeader {
      self.vals.get(T::HEADER_NAME).map(|v| T::from_str(&v))
   }
}

3. Do additional computation specific to the header type.

I think that would happen either: when setting the header value, or, when requesting the header value.

Given that, we might be able to do a slight twist on v2 to get something that is even more mirrored for getting and setting:

trait TypedHeader: FromStr + ToString {
    const HEADER_NAME: &'static str;

    fn set_value(value: impl Into<HeaderValue>) -> Result<(), Error>; // #2, #3.
    fn get_value() -> Result<&HeaderValue, Error>; // #2, #3.
}

impl Headers {
    fn get<T>(&self) -> Option<T>  where T: TypedHeader { // #1
       self.vals.get(T::HEADER_NAME).map(|v| T::from_str(&v))
   }

    fn set<T>(&mut self, header: T) where T: TypedHeader {
        self.vals
            .insert(T::HEADER_NAME.to_string(), header.to_string());
    }
}

Perhaps that's not quite flexible enough for get_value() and set_value() though, in terms of arguments and return type. There might be a better way.

[yoshuawuyts]

I've made a PR for another typed header impl in #203. I've also noted down the patterns we're using in http_types as part of this gist. The intent was never to have a separation between "typed headers" and regular headers, but instead use common method names + trait conversions to create a consistent API.

Header design overview

This is the current snapshot of the gist above. We're already using this API for http_types::security::ContentSecurityPolicy, and now also in #203. We need to adjust the unstable TraceContext headers to follow this API as well. What this API provides is:

• 3 constructors: new, from_headers, TryFrom<HeaderValue>.
• 4 conversions to header types: apply, name, value, ToHeaderValues.
• 4 trait impls: Debug, Clone, Eq, PartialEq.

Together these form a base template that should work for all typed header implementations.

use crate::headers::{HeaderName, HeaderValue, Headers, ToHeaderValues};

struct CustomHeader {}

impl CustomHeader {
    /// Create a new instance of `CustomHeader`.
    fn new() -> Self {
        todo!();
    }

    /// Create an instance of `CustomHeader` from a `Headers` instance.
    fn from_headers(headers: impl AsRef<Headers>) -> crate::Result<Option<Self>> {
        todo!();
    }

    /// Insert a `HeaderName` + `HeaderValue` pair into a `Headers` instance.
    fn apply(&self, headers: impl AsMut<Headers>) {
        todo!();
    }

    /// Get the `HeaderName`.
    fn name(&self) -> HeaderName {
        todo!();
    }

    /// Get the `HeaderValue`.
    fn value(&self) -> HeaderValue {
        todo!();
    }
}

// Conversion from `CustomHeader` -> `HeaderValue`.
impl ToHeaderValues for CustomHeader {
    type Iter;
    fn to_header_values(&self) -> crate::Result<Self::Iter> {
        todo!()
    }
}

[Fishrock123]

I think there's still a couple things missing here.

For example, from_headers() should probably return Option<Self>.

Also this doesn't really answer any questions regarding header value matching/verification...

I suppose for, say, content encoding matching you would do something like this:

if let Some(header) = ContentEncoding::from_headers(req) {
    if let Some(encoding) = header.accepts(&["br", "gzip"]) {
        // `encoding` is the preferred encoding here
    }
}

[yoshuawuyts]

Oops, yeah from_headers ought to probably return Result<Option<Self>> in order to catch both malformed requests and the absence of requests.

I'm not quite sure what the issue is with the example? I'm not sure I would nest if let Some statements when using it, but overall that looks like a reasonable API to me?

[ririsoft]

Oops, yeah from_headers ought to probably return Result<Option<Self>> in order to catch both malformed requests and the absence of requests.

While experiencing with such APIs on my side I wonder whether Option<Result<Self>> shouldn't be used instead ? The errors might occur only if the header exists after all. When using a Result<Option<Self>> I end with a lot more of Option::transpose or Result::transpose harness.

[Fishrock123]

I'm not sure I would nest if let Some statements when using it, but overall that looks like a reasonable API to me?

How would you write that?

Also if we are considering Option<Result<>> or similar, it's probably best to just use Result<> with a "none" error variant, similar to https://github.com/http-rs/tide/commit/377870615569ac01fbee589c01d69091a8c76de3 ?

[Fishrock123]

That being said on further thought maybe header errors should just be "none" unless inspected otherwise?

[Fishrock123]

Added to list:

• Referer
• User-Agent

[utherpally]

WWW-Authenticate header: From RFC 7235#appendix-A, the "realm" parameter is optional now.

[yoshuawuyts]

@langbamit thanks for reporting; I've filed https://github.com/http-rs/http-types/issues/280 to track it for the next major.

[yoshuawuyts]

Idempotentency-Key IETF draft. Perhaps we should implement this as a separate crate (submodule) until it stabilizes?

[Fishrock123]

IMO: implement draft headers under some compile time flag with unstable-like reduced guarantees.