Common HTTP request headers are unknown

[zakird]

From an HTTP request:

"request": {
  "url": {
    "scheme": "http",
    "host": "google.com",
    "path": "/"
  },
  "method": "GET",
  "headers": {
    "unknown": [
      {
        "key": "user_agent",
        "value": [
          "Mozilla/5.0 zgrab/0.x"
        ]
      },
      {
        "key": "accept",
        "value": [
          "*/*"
        ]
      }
    ]
  },
  "host": "google.com"

These shouldn't be unknown given that we send them with every request. We should find the list commonly sent headers and output all of them as recognized.

[dadrian]

We have the list, we just must not be using it for Request, only Response.

[zakird]

That or, it might also not have the common client-specified headers. 🤷‍♂️

[justinbastress]

It appears that this is the list we're currently using:

var knownHeaders = map[string]bool{
	"accept_patch":                true,
	"accept_ranges":               true,
	"access_control_allow_origin": true,
	"age":                         true,
	"allow":                       true,
	"alt_svc":                     true,
	"alternate_protocol":          true,
	"cache_control":               true,
	"connection":                  true,
	"content_disposition":         true,
	"content_encoding":            true,
	"content_language":            true,
	"content_length":              true,
	"content_location":            true,
	"content_md5":                 true,
	"content_range":               true,
	"content_security_policy":     true,
	"content_type":                true,
	"expires":                     true,
	"last_modified":               true,
	"link":                        true,
	"location":                    true,
	"p3p":                         true,
	"pragma":                      true,
	"proxy_agent":                 true,
	"proxy_authenticate":          true,
	"public_key_pins":             true,
	"referer":                     true,
	"refresh":                     true,
	"retry_after":                 true,
	"server":                      true,
	"set_cookie":                  true,
	"status":                      true,
	"strict_transport_security":   true,
	"trailer":                     true,
	"transfer_encoding":           true,
	"upgrade":                     true,
	"vary":                        true,
	"via":                         true,
	"warning":                     true,
	"www_authenticate":            true,
	"x_content_duration":          true,
	"x_content_security_policy":   true,
	"x_content_type_options":      true,
	"x_forwarded_for":             true,
	"x_frame_options":             true,
	"x_powered_by":                true,
	"x_real_ip":                   true,
	"x_ua_compatible":             true,
	"x_webkit_csp":                true,
	"x_xss_protection":            true,
}

So this confirms your suspicions, in that these seem to be geared towards response headers -- but it seems that knownHeaders is only used in one place, so it doesn't seem like adding additional values should be a problem.

As far as I can tell, the same issue exists in zgrab classic (80.http.get.headers.accept isn't showing up in auto-complete, and 80.http.get.headers.accept:* returns nothing)-- do we want to fix this there too?

[dadrian]

I'd say merge common client and server headers into one list, and use it for both. That way we'll tell if servers send client headers, etc.

[justinbastress]

See also https://github.com/zmap/zgrab/pull/322