vector icon indicating copy to clipboard operation
vector copied to clipboard

Request for parse_aws_cloudfront_log or parse_w3c_extended_logfile function

Open irvintim opened this issue 2 years ago • 2 comments

The documentation for parse_regex suggests opening a ticket to request a parse_* function be added for a log format that isn't already available.

My request is for AWS Cloudfront Logs, the format is W3C Extended Log File format. Which is made up of tab-delimited log lines with 2 "comment" lines prepended with "#" at the top of the file, one with a version # and one with a list of the fields. https://www.w3.org/TR/WD-logfile.html

And is further defined on this doc.: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/AccessLogs.html

e.g.:

#Version: 1.0
#Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken x-forwarded-for ssl-protocol ssl-cipher x-edge-response-result-type cs-protocol-version fle-status fle-encrypted-fields c-port time-to-first-byte x-edge-detailed-result-type sc-content-type sc-content-len sc-range-start sc-range-end
2023-02-25	06:06:53	LAX50-P3	429	129.146.75.29	GET	xxxx.cloudfront.net	/menu/guiw	200	-	Mozilla/5.0%20(Windows%20NT%2010.0)%20AppleWebKit/537.36%20(KHTML,%20like%20Gecko)%20Chrome/89.0.4389.114%20Safari/537.36	id=3&nsbrand=1&nsvpx=phpinfo&protocol=nonexistent.1337%22%3E	startupapp=st	LambdaGeneratedResponse	O0woGPklvWn5Hx7ahYw1uQdUBF-CrSBEl8EK-kgi4u-Tacc1df1Skg==	foo.bar.com	https	330	0.115	-	TLSv1.3	TLS_AES_128_GCM_SHA256	LambdaGeneratedResponse	HTTP/1.1	-	-	35024	0.115	LambdaGeneratedResponse	-	0	-	-
2023-02-25	06:06:54	LAX50-P3	434	129.146.75.29	GET	xxxx.cloudfront.net	/.//WEB-INF/web.xml	200	-	-	--	LambdaGeneratedResponse	0c6ffuHcpguIKuR1kOgR-fE0IRp2-qbuL_AqhtcrgIWSxHbKoN3jCA==	foo.bar.com	https	100	0.113	-	TLSv1.3	TLS_AES_128_GCM_SHA256	LambdaGeneratedResponse	HTTP/1.1	-	-	35032	0.113	LambdaGeneratedResponse-0

irvintim avatar Feb 25 '23 06:02 irvintim