goaccess
goaccess copied to clipboard
allinurl
Custom JSON format for Caddy log structure
hi allinurl I'm trying to use GoAccess to analyze my Caddy logs, but I'm having trouble creating a custom JSON format (using docker)
{"level":"info","ts":1624526415.449846,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"192.168.0.1","remote_port":"22","client_ip":"192.168.0.3","proto":"HTTP/1.1","method":"GET","host":"example.com","uri":"/","headers":{"Priority":["u=0, i"],"Sec-Ch-Ua":["\"Not)A;Brand\";v=\"99\", \"Brave\";v=\"127\", \"Chromium\";v=\"127\""],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-Proto":["https"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"],"Sec-Fetch-Site":["none"],"X-Forwarded-Host":["example.com"],"Sec-Fetch-User":["?1"],"Sec-Gpc":["1"],"X-Forwarded-For":["10.0.0.0"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 Chrome/127.0.0.0 Safari/537.36"],"Accept-Language":["en-US,en;q=0.9"]}},"bytes_read":0,"user_id":"","duration":0.001574238,"size":0,"status":401,"resp_headers":{"Server":["Caddy"],"Www-Authenticate":["Basic realm=\"restricted\""]}}
this is my goaccess.conf file for now
addr 0.0.0.0
port 7890
daemonize false
real-time-html true
log-format json={"level":"%e","ts":%T,"logger":"%v","msg":"%r","remote_ip":"$.request.remote_ip","remote_port":"$.request.remote_port","client_ip":"$.request.client_ip","proto":"$.request.proto","method":"$.request.method","host":"$.request.host","uri":"$.request.uri","bytes_read":%b,"user_id":"%e","duration":%D,"size":%b,"status":%s}, ignore-null
time-format %s
date-format %s
debug-file /var/log/goaccess/debug.log
log-file /var/log/caddy/access.log
output /var/www/goaccess/index.html
i get this errors
FILE: /var/log/caddy/access.log
2024-07-31 00:28:18 ==1== Parsed 2 lines producing the following errors:
2024-07-31 00:28:18 ==1==
2024-07-31 00:28:18 ==1== Token 'h-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-Proto":["https"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"],"Sec-Fetch-Site":["none"],"X-Forwarded-Host":["example.com"],"Sec-Fetch-User":["?1"],"Sec-Gpc":["1"],"X-Forwarded-For":["10.0.0.0"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 Chrome/127.0.0.0 Safari/537.36"],"Accept-Language":["en-US,en;q=0.9"]' doesn't match specifier '%s'
2024-07-31 00:28:18 ==1==
2024-07-31 00:28:18 ==1== Format Errors - Verify your log/date/time format
2024-07-31 00:50:40 [SETTING UP STORAGE cat /var/log/goaccess/debug.log] {0} @ {0/s}
2024-07-31 00:50:40 ==1== GoAccess - version 1.9.3 - Jul 22 2024 13:14:37
2024-07-31 00:50:40 ==1== Config file: /etc/goaccess/goaccess.conf
2024-07-31 00:50:40 ==1== https://goaccess.io - <[email protected]>
2024-07-31 00:50:40 ==1== Released under the MIT License.
2024-07-31 00:50:40 ==1==
2024-07-31 00:50:40 ==1== FILE: /var/log/caddy/access.log
2024-07-31 00:50:40 ==1== Parsed 2 lines producing the following errors:
2024-07-31 00:50:40 ==1==
2024-07-31 00:50:40 ==1== A valid date is required.
2024-07-31 00:50:40 ==1== A valid date is required.
2024-07-31 00:50:40 ==1==
2024-07-31 00:50:40 ==1== Format Errors - Verify your log/date/time format
2024-07-31 00:59:12 [SETTING UP STORAGE cat /var/log/goaccess/debug.log] {0} @ {0/s}
2024-07-31 00:59:12 ==1== GoAccess - version 1.9.3 - Jul 22 2024 13:14:37
2024-07-31 00:59:12 ==1== Config file: /etc/goaccess/goaccess.conf
2024-07-31 00:59:12 ==1== https://goaccess.io - <[email protected]>
2024-07-31 00:59:12 ==1== Released under the MIT License.
2024-07-31 00:59:12 ==1==
2024-07-31 00:59:12 ==1== FILE: /bin/sh
2024-07-31 00:59:12 ==1== Parsed 10 lines producing the following errors:
2024-07-31 00:59:12 ==1==
2024-07-31 00:59:12 ==1== Incompatible format due to early parsed line ending '\0'.
2024-07-31 00:59:12 ==1== Incompatible format due to early parsed line ending '\0'.
2024-07-31 00:59:12 ==1== Incompatible format due to early parsed line ending '\0'.
2024-07-31 00:59:12 ==1== Incompatible format due to early parsed line ending '\0'.
2024-07-31 00:59:12 ==1== Incompatible format due to early parsed line ending '\0'.
2024-07-31 00:59:12 ==1== Incompatible format due to early parsed line ending '\0'.
2024-07-31 00:59:12 ==1== Incompatible format due to early parsed line ending '\0'.
2024-07-31 00:59:12 ==1== Incompatible format due to early parsed line ending '\0'.
2024-07-31 00:59:12 ==1== Incompatible format due to early parsed line ending '\0'.
2024-07-31 00:59:12 ==1== Incompatible format due to early parsed line ending '\0'.
2024-07-31 00:59:12 ==1==
2024-07-31 00:59:12 ==1== Format Errors - Verify your log/date/time format
This should do it:
# goaccess access.log --log-format=CADDY --date-spec=min
I updated goaccess.conf as follows
addr 0.0.0.0
port 7890
daemonize false
real-time-html true
log-format CADDY
date-spec min
debug-file /var/log/goaccess/debug.log
log-file /var/log/caddy/access.log
output /var/www/goaccess/index.html
it gives me this error
2024-07-31 06:28:49 [SETTING UP STORAGE cat /var/log/goaccess/debug.log] {0} @ {0/s}
2024-07-31 06:28:49 ==1== GoAccess - version 1.9.3 - Jul 22 2024 13:14:37
2024-07-31 06:28:49 ==1== Config file: /etc/goaccess/goaccess.conf
2024-07-31 06:28:49 ==1== https://goaccess.io - <[email protected]>
2024-07-31 06:28:49 ==1== Released under the MIT License.
2024-07-31 06:28:49 ==1==
2024-07-31 06:28:49 ==1== FILE: /bin/sh
2024-07-31 06:28:49 ==1== Parsed 10 lines producing the following errors:
2024-07-31 06:28:49 ==1==
2024-07-31 06:28:49 ==1== IPv4/6 is required.
2024-07-31 06:28:49 ==1== IPv4/6 is required.
2024-07-31 06:28:49 ==1== IPv4/6 is required.
2024-07-31 06:28:49 ==1== IPv4/6 is required.
2024-07-31 06:28:49 ==1== IPv4/6 is required.
2024-07-31 06:28:49 ==1== IPv4/6 is required.
2024-07-31 06:28:49 ==1== IPv4/6 is required.
2024-07-31 06:28:49 ==1== IPv4/6 is required.
2024-07-31 06:28:49 ==1== IPv4/6 is required.
2024-07-31 06:28:49 ==1== IPv4/6 is required.
2024-07-31 06:28:49 ==1==
2024-07-31 06:28:49 ==1== Format Errors - Verify your log/date/time format
It looks like the first 10 lines of your JSON log may not be valid. Could you please share the first 20 lines directly from your access.log?
these are the same logs that I provided earlier basically. I don't know if I'm doing anything in particular wrong. In goaccess.conf file, i have specified the log format as CADDY, which is not compatible with the JSON log format produced by Caddy (i think).
{"level":"info","ts":1722377868.638059,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"XXX.XXX.XXX.XXX","remote_port":"1234","client_ip":"XXX.XXX.XXX.XXX","proto":"HTTP/1.1","method":"GET","host":"XXXXXXX","uri":"/","headers":{"Priority":["u=0, i"],"Sec-Ch-Ua":["\"Not)A;Brand\";v=\"99\", \"Brave\";v=\"127\", \"Chromium\";v=\"127\""],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-Proto":["https"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"],"Sec-Fetch-Site":["none"],"X-Forwarded-Host":["XXXXXXX"],"Sec-Fetch-User":["?1"],"Sec-Gpc":["1"],"X-Forwarded-For":["XXX.XXX.XXX.XXX"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Upgrade-Insecure-Requests":["1"],"User-Agent":["XXXXXXX"],"Accept-Language":["en-US,en;q=0.9"]}},"bytes_read":0,"user_id":"","duration":0.001574238,"size":0,"status":401,"resp_headers":{"Server":["Caddy"],"Www-Authenticate":["Basic realm=\"restricted\""]}}
{"level":"info","ts":1722377884.6235218,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"XXX.XXX.XXX.XXX","remote_port":"1234","client_ip":"XXX.XXX.XXX.XXX","proto":"HTTP/1.1","method":"GET","host":"XXXXXXX","uri":"/","headers":{"Sec-Ch-Ua":["\"Not)A;Brand\";v=\"99\", \"Brave\";v=\"127\", \"Chromium\";v=\"127\""],"Sec-Fetch-Mode":["navigate"],"Upgrade-Insecure-Requests":["1"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Sec-Gpc":["1"],"X-Forwarded-For":["XXX.XXX.XXX.XXX"],"X-Forwarded-Proto":["https"],"Cache-Control":["max-age=0"],"Sec-Ch-Ua-Mobile":["?0"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Site":["none"],"Priority":["u=0, i"],"Sec-Fetch-User":["?1"],"X-Forwarded-Host":["XXXXXXX"],"User-Agent":["XXXXXXX"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"],"Accept-Encoding":["gzip, deflate, br, zstd"]}},"bytes_read":0,"user_id":"","duration":0.000091731,"size":0,"status":401,"resp_headers":{"Server":["Caddy"],"Www-Authenticate":["Basic realm=\"restricted\""]}}
{"level":"info","ts":1722399905.8303173,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"XXX.XXX.XXX.XXX","remote_port":"1234","client_ip":"XXX.XXX.XXX.XXX","proto":"HTTP/1.1","method":"GET","host":"XXXXXXX","uri":"/","headers":{"Sec-Fetch-Dest":["document"],"X-Forwarded-For":["XXX.XXX.XXX.XXX"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"],"Priority":["u=0, i"],"Sec-Ch-Ua":["\"Not)A;Brand\";v=\"99\", \"Brave\";v=\"127\", \"Chromium\";v=\"127\""],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Sec-Gpc":["1"],"User-Agent":["XXXXXXX"],"Accept-Encoding":["gzip, deflate, br, zstd"],"X-Forwarded-Proto":["https"],"Upgrade-Insecure-Requests":["1"],"X-Forwarded-Host":["XXXXXXX"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Site":["none"]}},"bytes_read":0,"user_id":"","duration":0.00088854,"size":0,"status":401,"resp_headers":{"Server":["Caddy"],"Www-Authenticate":["Basic realm=\"restricted\""]}}
{"level":"info","ts":1722452074.5979362,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"XXX.XXX.XXX.XXX","remote_port":"XXXX","client_ip":"XXX.XXX.XXX.XXX","proto":"HTTP/1.1","method":"GET","host":"anonymized-host","uri":"/favicon.ico","headers":{"X-Forwarded-Host":["XXXXXXX"],"X-Forwarded-Proto":["https"],"User-Agent":["XXXXXXX"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Dest":["image"],"X-Forwarded-For":["XXX.XXX.XXX.XXX"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-Mode":["no-cors"],"Sec-Fetch-Site":["same-origin"],"Accept":["image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"],"Authorization":["REDACTED"],"Priority":["u=1, i"],"Referer":["XXXXXXX"],"Sec-Gpc":["1"],"If-Modified-Since":["Mon, 14 Feb 2022 05:51:54 GMT"],"If-None-Match":["W/\"47e-17ef6c99890\""],"Sec-Ch-Ua":["\"Not)A;Brand\";v=\"99\", \"Brave\";v=\"127\", \"Chromium\";v=\"127\""],"Sec-Ch-Ua-Platform":["\"Windows\""]}},"bytes_read":0,"user_id":"root","duration":0.000941437,"size":0,"status":304,"resp_headers":{"X-Xss-Protection":["1; mode=block"],"Etag":["W/\"47e-17ef6c99890\""],"Date":["Wed, 31 Jul 2024 18:54:34 GMT"],"Content-Security-Policy":["default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; img-src 'self' data:; font-src 'self'; frame-ancestors 'self'; base-uri 'self'; form-action 'self';"],"Accept-Ranges":["bytes"],"Cache-Control":["public, max-age=0"],"Last-Modified":["Mon, 14 Feb 2022 05:51:54 GMT"],"Referrer-Policy":["strict-origin-when-cross-origin"],"Strict-Transport-Security":["max-age=31536000; includeSubDomains; preload"],"X-Content-Type-Options":["nosniff"],"X-Frame-Options":["SAMEORIGIN"],"Vary":["Accept-Encoding"]}}
{"level":"info","ts":1722377868.638059,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"XXX.XXX.XXX.XXX","remote_port":"XXXX","client_ip":"XXX.XXX.XXX.XXX","proto":"HTTP/1.1","method":"GET","host":"XXX","uri":"/","headers":{"Priority":["u=0, i"],"Sec-Ch-Ua":[""Not)A;Brand";v="99", "Brave";v="127", "Chromium";v="127""],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-Proto":["https"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8"],"Sec-Fetch-Site":["none"],"X-Forwarded-Host":["XXX"],"Sec-Fetch-User":["?1"],"Sec-Gpc":["1"],"X-Forwarded-For":["XXX.XXX.XXX.XXX"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Ch-Ua-Platform":[""Windows""],"Upgrade-Insecure-Requests":["1"],"User-Agent":["anonymized-user-agent"],"Accept-Language":["en-US,en;q=0.9"]}},"bytes_read":0,"user_id":"","duration":0.001574238,"size":0,"status":401,"resp_headers":{"Server":["Caddy"],"Www-Authenticate":["Basic realm="restricted""]}}
{"level":"info","ts":1722377884.6235218,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"XXX.XXX.XXX.XXX","remote_port":"XXXX","client_ip":"XXX.XXX.XXX.XXX","proto":"HTTP/1.1","method":"GET","host":"XXX","uri":"/","headers":{"Sec-Ch-Ua":[""Not)A;Brand";v="99", "Brave";v="127", "Chromium";v="127""],"Sec-Fetch-Mode":["navigate"],"Upgrade-Insecure-Requests":["1"],"Sec-Ch-Ua-Platform":[""Windows""],"Sec-Gpc":["1"],"X-Forwarded-For":["XXX.XXX.XXX.XXX"],"X-Forwarded-Proto":["https"],"Cache-Control":["max-age=0"],"Sec-Ch-Ua-Mobile":["?0"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Site":["none"],"Priority":["u=0, i"],"Sec-Fetch-User":["?1"],"X-Forwarded-Host":["XXX"],"User-Agent":["anonymized-user-agent"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8"],"Accept-Encoding":["gzip, deflate, br, zstd"]}},"bytes_read":0,"user_id":"","duration":0.000091731,"size":0,"status":401,"resp_headers":{"Server":["Caddy"],"Www-Authenticate":["Basic realm="restricted""]}}
{"level":"info","ts":1722399905.8303173,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"XXX.XXX.XXX.XXX","remote_port":"XXXX","client_ip":"XXX.XXX.XXX.XXX","proto":"HTTP/1.1","method":"GET","host":"XXX","uri":"/","headers":{"Sec-Fetch-Dest":["document"],"X-Forwarded-For":["XXX.XXX.XXX.XXX"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8"],"Priority":["u=0, i"],"Sec-Ch-Ua":[""Not)A;Brand";v="99", "Brave";v="127", "Chromium";v="127""],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":[""Windows""],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Sec-Gpc":["1"],"User-Agent":["anonymized-user-agent"],"Accept-Encoding":["gzip, deflate, br, zstd"],"X-Forwarded-Proto":["https"],"Upgrade-Insecure-Requests":["1"],"X-Forwarded-Host":["XXX"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Site":["none"]}},"bytes_read":0,"user_id":"","duration":0.00088854,"size":0,"status":401,"resp_headers":{"Server":["Caddy"],"Www-Authenticate":["Basic realm="restricted""]}}
maybe i need to put in the goaccess.conf file something like this
addr 0.0.0.0
port 7890
daemonize false
real-time-html true
log-format "%h %^[%d:%t %^] \"%r\" %s %b \"%u\" \"%H\" \"%R\""
date-spec %d:%t
debug-file /var/log/goaccess/debug.log
log-file /var/log/caddy/access.log
output /var/www/goaccess/index.html
That same command using the CADDY log format works for me.
Please try using --no-global-config and try:
# goaccess access.log --log-format=CADDY --date-spec=min
i change goaccess.conf with this
addr 0.0.0.0
port 7890
daemonize false
real-time-html true
no-global-config true
log-format CADDY
date-spec min
debug-file /var/log/goaccess/debug.log
log-file /var/log/caddy/access.log
output /var/www/goaccess/index.html
and still give me the same error, at this point i really don't know
2024-08-04 18:18:06 [SETTING UP STORAGE cat /var/log/goaccess/debug.log] {0} @ {0/s}
2024-08-04 18:18:06 ==1== GoAccess - version 1.9.3 - Jul 22 2024 13:14:37
2024-08-04 18:18:06 ==1== Config file: /etc/goaccess/goaccess.conf
2024-08-04 18:18:06 ==1== https://goaccess.io - <[email protected]>
2024-08-04 18:18:06 ==1== Released under the MIT License.
2024-08-04 18:18:06 ==1==
2024-08-04 18:18:06 ==1== FILE: /bin/sh
2024-08-04 18:18:06 ==1== Parsed 10 lines producing the following errors:
2024-08-04 18:18:06 ==1==
2024-08-04 18:18:06 ==1== IPv4/6 is required.
2024-08-04 18:18:06 ==1== IPv4/6 is required.
2024-08-04 18:18:06 Cleaning up resources...
2024-08-04 18:18:06 ==1== IPv4/6 is required.
2024-08-04 18:18:06 ==1== IPv4/6 is required.
2024-08-04 18:18:06 ==1== IPv4/6 is required.
2024-08-04 18:18:06 ==1== IPv4/6 is required.
2024-08-04 18:18:06 ==1== IPv4/6 is required.
2024-08-04 18:18:06 ==1== IPv4/6 is required.
2024-08-04 18:18:06 ==1== IPv4/6 is required.
2024-08-04 18:18:06 ==1== IPv4/6 is required.
2024-08-04 18:18:06 ==1==
2024-08-04 18:18:06 ==1== Format Errors - Verify your log/date/time format
2024-08-04 18:18:07 [SETTING UP STORAGE cat /var/log/goaccess/debug.log] {0} @ {0/s}
I am also getting this IPv4/6 is required issue with Caddy, even with --date-spec=min.
Are you still having trouble with this? Just to rule out any config file issues, try running GoAccess directly with:
# goaccess access.log --log-format=CADDY --date-spec=min
If that doesn't work, @makew0rld feel free to share a few lines from your access log and I'll take a closer look!