goaccess icon indicating copy to clipboard operation
goaccess copied to clipboard

Custom log format - using port as protocol

Open ds-cbo opened this issue 2 months ago • 5 comments

My nginx log format is this:

log_format logger-json escape=json '{'
     '"remote_addr": "$remote_addr", "remote_user": "$remote_user"'
     ', "time_iso8601": "$time_iso8601", "msec": $msec'
     ', "host": "$http_host", "method": "$request_method", "uri": "$request_uri"'
     ', "status": "$status", "body_bytes_sent": "$body_bytes_sent"'
     ', "referer": "$http_referer", "user_agent": "$http_user_agent", "server_name": "$server_name"'
     ', "port": "$server_port", "ut": "$upstream_response_time", "rl": "$request_length"'
     ', "ssl_client_s_dn": "$ssl_client_s_dn"'
     ', "ssl_cipher": "$ssl_cipher", "ssl_protocol": "$ssl_protocol"'
     ', "rt": "$request_time", "uct": "$upstream_connect_time"'
     '}';

Which I have converted to goaccess configuration as

time-format %H:%M:%S
date-format %Y-%m-%d

log-format {"remote_addr": "%h", "remote_user": "%e", "time_iso8601": "%dT%t+%^", "msec": %^, "host": "%^", "method": "%m", "uri": "%U", "status": "%s", "body_bytes_sent": "%b", "referer": "%R", "user_agent": "%u", "server_name": "%v", "port": "%H", "ut": "%^", "rl": "%^", "ssl_client_s_dn": "%^", "ssl_cipher": "%k", "ssl_protocol": "%K", "rt": "%T", "uct": "%^"}

However, this is not fully accepted:

Token '80' doesn't match specifier '%H'

Would it be possible to allow %H to accept 80 for HTTP and 443 for HTTPS ?

--

And semi-related, I'm surprised there seems to be no specifier for request_length, but that might warrant its own ticket

ds-cbo avatar Oct 23 '25 08:10 ds-cbo

Would you be able to post a few sample lines from your log so I can take a look? %H is for the actual protocol so it won't accept the port.

allinurl avatar Oct 26 '25 00:10 allinurl

Sure:

{"remote_addr": "123.123.123.123", "remote_user": "", "time_iso8601": "2025-10-27T06:26:36+01:00", "msec": 1761542796.296, "host": "example.net", "method": "GET", "uri": "/", "status": "404", "body_bytes_sent": "146", "referer": "", "user_agent": "", "server_name": "example.net", "port": "80", "ut": "", "rl": "59", "ssl_client_s_dn": "", "ssl_cipher": "", "ssl_protocol": "", "rt": "0.000", "uct": ""}
{"remote_addr": "123.123.123.123", "remote_user": "", "time_iso8601": "2025-10-22T19:14:29+02:00", "msec": 1761153269.557, "host": "example.net", "method": "GET", "uri": "/", "status": "200", "body_bytes_sent": "24506", "referer": "", "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.3", "server_name": "example.net", "port": "443", "ut": "", "rl": "218", "ssl_client_s_dn": "", "ssl_cipher": "TLS_AES_256_GCM_SHA384", "ssl_protocol": "TLSv1.3", "rt": "0.000", "uct": ""}

I thought the protocol would be HTTP/HTTPS, maybe the naming is a bit confusing then

ds-cbo avatar Oct 28 '25 10:10 ds-cbo

whoops, wrong button, could you re-open?

ds-cbo avatar Oct 28 '25 10:10 ds-cbo

Good point. HTTP/HTTPS refers to the protocol (secure or not), while HTTP/1.0, 1.1, 2, and 3 are versions of that protocol. We could add an option to accept that instead? or if you don't need to link the request to the protocol directly, you could use %e for the port to reuse the user panel.

allinurl avatar Oct 29 '25 21:10 allinurl

I am already using %e for remote_user, but I could use %M instead of for mime-type, it's a bit wonky but will do for now. Thanks

ds-cbo avatar Oct 30 '25 08:10 ds-cbo