GoAccess and AWS WAF access.logs (web application firewall)

[mandys]

Could this be used to parse aws web application firewall logs ? Those are in Json format and in display we require requests that were blocked by firewall.

[allinurl]

Do you have any sample log lines that I can take a look? Thanks

[mandys]

This is one line in the log where the WAF has blocked a bad request.

{
	"timestamp": 1626141957554,
	"formatVersion": 1,
	"webaclId": "arn:aws:wafv2:us-west-2:912345678905:regional/webacl/WebACL/12fa5cbd-4fdc-4c3e-94cd-8e123f8fd123",
	"terminatingRuleId": "AWS-AWSManagedRulesCommonRuleSet",
	"terminatingRuleType": "MANAGED_RULE_GROUP",
	"action": "BLOCK",
	"terminatingRuleMatchDetails": [],
	"httpSourceName": "ALB",
	"httpSourceId": "912345678905-app/premium/fd1f12faadfc1d12",
	"ruleGroupList": [{
		"ruleGroupId": "AWS#AWSManagedRulesCommonRuleSet",
		"terminatingRule": {
			"ruleId": "UserAgent_BadBots_HEADER",
			"action": "BLOCK",
			"ruleMatchDetails": null
		},
		"nonTerminatingMatchingRules": [],
		"excludedRules": null
	}],
	"rateBasedRuleList": [],
	"nonTerminatingMatchingRules": [],
	"requestHeadersInserted": null,
	"responseCodeSent": null,
	"httpRequest": {
		"clientIp": "54.36.149.8",
		"country": "FR",
		"headers": [{
			"name": "Host",
			"value": "goaccess.io"
		}, {
			"name": "User-Agent",
			"value": "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"
		}, {
			"name": "Accept",
			"value": "*/*"
		}, {
			"name": "Accept-Encoding",
			"value": "deflate, gzip, br"
		}],
		"uri": "/robots.txt",
		"args": "",
		"httpVersion": "HTTP/1.1",
		"httpMethod": "GET",
		"requestId": "1-60ecf505-5eba40c41bfaa68602949dd8"
	}
}

[mandys]

This is an allowed request

{
	"timestamp": 1626141961811,
	"formatVersion": 1,
	"webaclId": "arn:aws:wafv2:us-west-2: 912345678905:regional/webacl/WebACL/12fa5cbd-4fdc-4c3e-94cd-8e123f8fd123",
	"terminatingRuleId": "Default_Action",
	"terminatingRuleType": "REGULAR",
	"action": "ALLOW",
	"terminatingRuleMatchDetails": [],
	"httpSourceName": "ALB",
	"httpSourceId": "912345678905-app/premium/fd1f12faadfc1d12",
	"ruleGroupList": [{
		"ruleGroupId": "AWS#AWSManagedRulesCommonRuleSet",
		"terminatingRule": null,
		"nonTerminatingMatchingRules": [],
		"excludedRules": null
	}, {
		"ruleGroupId": "AWS#AWSManagedRulesSQLiRuleSet",
		"terminatingRule": null,
		"nonTerminatingMatchingRules": [],
		"excludedRules": null
	}, {
		"ruleGroupId": "AWS#AWSManagedRulesLinuxRuleSet",
		"terminatingRule": null,
		"nonTerminatingMatchingRules": [],
		"excludedRules": null
	}, {
		"ruleGroupId": "AWS#AWSManagedRulesPHPRuleSet",
		"terminatingRule": null,
		"nonTerminatingMatchingRules": [],
		"excludedRules": null
	}, {
		"ruleGroupId": "AWS#AWSManagedRulesKnownBadInputsRuleSet",
		"terminatingRule": null,
		"nonTerminatingMatchingRules": [],
		"excludedRules": null
	}],
	"rateBasedRuleList": [],
	"nonTerminatingMatchingRules": [],
	"requestHeadersInserted": null,
	"responseCodeSent": null,
	"httpRequest": {
		"clientIp": "23.239.28.171",
		"country": "US",
		"headers": [{
			"name": "Host",
			"value": "www.goaccess.io"
		}, {
			"name": "Accept",
			"value": "*/*"
		}, {
			"name": "Accept-Encoding",
			"value": "gzip"
		}, {
			"name": "User-Agent",
			"value": "Better Uptime Bot Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"
		}],
		"uri": "/",
		"args": "",
		"httpVersion": "HTTP/1.1",
		"httpMethod": "GET",
		"requestId": "1-60ecf509-020da2052161001849f5658a"
	}
}

[allinurl]

Sorry for late reply. This should do it:

goaccess access.log --log-format='{ "timestamp": "%x", "httpRequest": { "clientIp": "%h", "headers": [{"name": "Host", "value": "%v"}, {"name":"User-Agent", "value": "%u"}], "uri": "%U", "args": "%q", "httpVersion": "%H", "httpMethod": "%m" } }' --date-format=%* --time-format=%*

[allinurl]

Actually you can use %e for the action, e.g., BLOCKED, ALLOWED, etc

goaccess access.log --log-format='{ "timestamp": "%x", "action": "%e", "httpRequest": { "clientIp": "%h", "headers": [{"name": "Host", "value": "%v"}, {"name":"User-Agent", "value": "%u"}], "uri": "%U", "args": "%q", "httpVersion": "%H", "httpMethod": "%m" } }' --date-format=%* --time-format=%*

[mandys]

Thank you for getting back. I tried

/usr/local/bin/goaccess aws-waf-log.txt goaccess access.log --log-format='{ "timestamp": "%x", "action": "%e", "httpRequest": { "clientIp": "%h", "headers": [{"name": "Host", "value": "%v"}, {"name":"User-Agent", "value": "%u"}], "uri": "%U", "args": "%q", "httpVersion": "%H", "httpMethod": "%m" } }' --date-format=%* --time-format=%*

Parsed 1 lines producing the following errors:
Token '6141873127,' doesn't match specifier '%x'
Format Errors - Verify your log/date/time format

Then changed timestamp to %f and tried again

/usr/local/bin/goaccess aws-waf-log.txt goaccess access.log --log-format='{ "timestamp": "%f", "action": "%e", "httpRequest": { "clientIp": "%h", "headers": [{"name": "Host", "value": "%v"}, {"name":"User-Agent", "value": "%u"}], "uri": "%U", "args": "%q", "httpVersion": "%H", "httpMethod": "%m" } }' --date-format=%* --time-format=%*

Parsed 1 lines producing the following errors:
Token 'west-2: 912345678905:regional/webacl/WebACL/60fa5cbd-4fdc-4c3e-94cd-8e540f8fd189' doesn't match specifier '%h'
Format Errors - Verify your log/date/time format

Still doesn't work.

Sample

{
  "timestamp": 1626141873127,
  "formatVersion": 1,
  "webaclId": "arn:aws:wafv2:us-west-2: 912345678905:regional/webacl/WebACL/60fa5cbd-4fdc-4c3e-94cd-8e540f8fd189",
  "terminatingRuleId": "Default_Action",
  "terminatingRuleType": "REGULAR",
  "action": "ALLOW",
  "terminatingRuleMatchDetails": [],
  "httpSourceName": "ALB",
  "httpSourceId": "912345678905-app/cname/fd6f11faadfc8d83",
  "ruleGroupList": [
    {
      "ruleGroupId": "AWS#AWSManagedRulesCommonRuleSet",
      "terminatingRule": null,
      "nonTerminatingMatchingRules": [],
      "excludedRules": null
    },
    {
      "ruleGroupId": "AWS#AWSManagedRulesSQLiRuleSet",
      "terminatingRule": null,
      "nonTerminatingMatchingRules": [],
      "excludedRules": null
    },
    {
      "ruleGroupId": "AWS#AWSManagedRulesLinuxRuleSet",
      "terminatingRule": null,
      "nonTerminatingMatchingRules": [],
      "excludedRules": null
    },
    {
      "ruleGroupId": "AWS#AWSManagedRulesPHPRuleSet",
      "terminatingRule": null,
      "nonTerminatingMatchingRules": [],
      "excludedRules": null
    },
    {
      "ruleGroupId": "AWS#AWSManagedRulesKnownBadInputsRuleSet",
      "terminatingRule": null,
      "nonTerminatingMatchingRules": [],
      "excludedRules": null
    }
  ],
  "rateBasedRuleList": [],
  "nonTerminatingMatchingRules": [],
  "requestHeadersInserted": null,
  "responseCodeSent": null,
  "httpRequest": {
    "clientIp": "170.178.178.163",
    "country": "US",
    "headers": [
      {
        "name": "host",
        "value": "goaccess.io"
      },
      {
        "name": "upgrade-insecure-requests",
        "value": "1"
      },
      {
        "name": "user-agent",
        "value": "Mozilla/5.0 (Linux; Android 11; SM-G9810 Build/RP1A.200720.012; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/89.0.4389.105 Mobile Safari/537.36"
      },
      {
        "name": "accept",
        "value": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
      },
      {
        "name": "x-requested-with",
        "value": "net.radio"
      },
      {
        "name": "sec-fetch-site",
        "value": "none"
      },
      {
        "name": "sec-fetch-mode",
        "value": "navigate"
      },
      {
        "name": "sec-fetch-user",
        "value": "?1"
      },
      {
        "name": "sec-fetch-dest",
        "value": "document"
      },
      {
        "name": "accept-encoding",
        "value": "gzip, deflate"
      },
      {
        "name": "accept-language",
        "value": "en-GB,en;q=0.9,zh-CN;q=0.8,zh;q=0.7,en-US;q=0.6"
      },
      {
        "name": "cookie",
        "value": "__utmc=121975063; __utmz=121975063.1596376219.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); "
      }
    ],
    "uri": "/s/landing/abcd",
    "args": "vtg_device=android&vtg_webview=1",
    "httpVersion": "HTTP/2.0",
    "httpMethod": "GET",
    "requestId": "1-71ecf4b1-0fb74c2e5e00cfc46a080aab"
  }
}

[allinurl]

Are you running v1.5.1 (latest)? Here's what I get for that last line you posted:

[mandys]

Just updated to 1.5.1 ( I was on 1.4 ). Code provided by you is working great!

I am able to see the block vs allow percentages which is helpful.

But, if I want to dig down a bit into which requests were blocked, would that be possible in any release ?

Currently, I don't see any waf parser and then only way is to use amazon athena, kinesis, kibana dashboard etc and launch a complex setup.

goaccess has a great opportunity to become a one stop shop.

Thanks for all the hardwork!

[allinurl]

I'm working on #117 as we speak, so you should be able to simply filter/search by whatever request you are interested to see and should display the relevant stats. So this is coming for sure! Also, on there is #1202 which would add much more flexibility to custom fields and panels. That would be the second biggest feature goaccess is getting.

My question to you, are there specific fields you are interested in from the JSON request you posted above?