Skip hyphen values (i.e. no value is marked with '-')

[ketodiet]

This is more of a question than an issue or feature request.

In our IIS logs a non-value is marked with '-', see example below:

2018-11-19 13:00:38 W3SVC1 172.30.0.176 GET /keto-app/cleanup-02.min.css - 443 - 73.18.227.138 Mozilla/5.0+(iPhone;+CPU+iPhone+OS+12_1+like+Mac+OS+X)+AppleWebKit/605.1.15+(KHTML,+like+Gecko) https://www.lowcarbmaven.com/keto-chocolate-chip-cookies/?utm_source=keto-app&utm_medium=referral 200 0 0 46

For example the hyphen just before 443 in the line above is the query string. When marked with '-' it simply means there is no query string for the given request.

Is there a way to handle this correctly in goaccess or do I need to pre-process and clean-up the logs before passing them to goaccess?

Also is there a way to translate 443 to HTTPS and 80 to HTTP ?

Here's the config for the IIS logs:

log-format %d %t %^ %^ %m %U %^ %^ %^ %h %u %R %s %^ %^ %L date-format %Y-%m-%d time-format %H:%M:%S

Thank you

[allinurl]

Currently you would need to preprocess the log. I can add either an option to ignore the dash or have it always ignore it if there's nothing else. Would that be a safe assumption?

Are you looking to report HTTP and HTTPS on its own panel?

[bodrick]

So just wanted to add that I also have this problem and it would be good to have some sort of option to treat the dash as empty or something along those lines, or even have the ability to do some simple logic or regex but I see that as more complications.

[JeepNL]

I'm using .NET 8 Preview 7 and Microsoft's Kestrel Web Server with W3CLogging enabled (See: builder.Services.AddW3CLogging(options =>) on an Ubuntu 22.04 VPS and if there's no query string it writes the a - to the log file.

If it's possible I would like an option to ignore the - (for any field, and possibly a char you can define yourself) because my console (without a query string) looks now like this:

[allinurl]

@JeepNL Could you share a few lines from your access log along with the format you're using? This would help me examine it more effectively.

[JeepNL]

ASP.NET Core W3CLogger for Kestrel Web Server creates a space delimited log file.

Sample: (I've redacted some info with ***)

#Version: 1.0
#Start-Date: 2023-09-06 22:24:35
#Fields: date time c-ip s-port cs-method cs-uri-stem cs-uri-query sc-status time-taken cs-version cs-host
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /css/app.min.css - 200 5.7776 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/blazor.webassembly.js - 200 8.3352 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /Blazor.Ictz.Client.styles.css - 200 9.5714 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /css/bootstrap/bootstrap.min.css - 200 27.8611 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /css/open-iconic/font/css/open-iconic-bootstrap.min.css - 200 3.4737 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/dotnet.js - 200 6.2192 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/blazor.boot.json - 200 3.4001 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/dotnet.runtime.8.0.0-preview.7.23375.6.dig1gj2ycu.js - 200 4.6011 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/dotnet.native.8.0.0-preview.7.23375.6.yim8jl7plt.js - 200 4.984 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/dotnet.native.wasm - 200 42.3729 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/Blazor.Ictz.Client.wasm - 200 2.8015 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/Microsoft.Extensions.Configuration.wasm - 200 8.7517 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/Microsoft.Extensions.Configuration.Json.wasm - 200 8.9023 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/Microsoft.Extensions.Configuration.Abstractions.wasm - 200 9.0267 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/Microsoft.AspNetCore.Metadata.wasm - 200 9.1313 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/Microsoft.Extensions.DependencyInjection.Abstractions.wasm - 200 10.5769 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/Microsoft.AspNetCore.Components.Web.wasm - 200 11.1459 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/Microsoft.AspNetCore.Components.wasm - 200 14.7515 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/Microsoft.AspNetCore.Components.Forms.wasm - 200 15.9316 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/Microsoft.AspNetCore.Authorization.wasm - 200 16.1666 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/Microsoft.AspNetCore.Components.Authorization.wasm - 200 16.1256 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/System.ComponentModel.wasm - 200 9.2712 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/System.ComponentModel.Primitives.wasm - 200 9.6231 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/System.Collections.Specialized.wasm - 200 10.0293 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/Microsoft.JSInterop.WebAssembly.wasm - 200 10.4033 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/Microsoft.Extensions.Primitives.wasm - 200 10.6628 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/System.Linq.wasm - 200 9.7393 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/Blazor.Ictz.Shared.wasm - 200 24.38 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/System.Diagnostics.DiagnosticSource.wasm - 200 10.79 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/System.ComponentModel.TypeConverter.wasm - 200 11.0794 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/System.ComponentModel.Annotations.wasm - 200 11.3702 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/System.Collections.wasm - 200 11.5229 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/System.Collections.Concurrent.wasm - 200 11.8128 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/Microsoft.JSInterop.wasm - 200 12.2996 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/Microsoft.Extensions.Options.wasm - 200 12.4922 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/Microsoft.Extensions.DependencyInjection.wasm - 200 12.9245 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/Microsoft.Extensions.Logging.Abstractions.wasm - 200 12.7322 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/Microsoft.AspNetCore.Components.WebAssembly.wasm - 200 24.9698 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/System.ObjectModel.wasm - 200 6.2377 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/System.Net.Primitives.wasm - 200 6.3463 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/System.Runtime.wasm - 200 6.9504 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/System.wasm - 200 6.9129 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/System.Collections.NonGeneric.wasm - 200 19.5172 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/Microsoft.Extensions.Logging.wasm - 200 25.5562 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/System.Memory.wasm - 200 14.582 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/System.Net.Http.Json.wasm - 200 14.654 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/System.Runtime.InteropServices.JavaScript.wasm - 200 14.3759 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/System.Security.Claims.wasm - 200 14.3209 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/System.Text.Encodings.Web.wasm - 200 14.2843 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/System.Private.Uri.wasm - 200 14.6692 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/System.Linq.Expressions.wasm - 200 24.9072 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/System.Net.Http.wasm - 200 15.1441 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/System.Text.Json.wasm - 200 19.0529 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/System.Text.RegularExpressions.wasm - 200 19.0422 HTTP/2 ***.net
2023-09-07 07:02:35 ::ffff:195.154.***.*** 443 GET /_framework/System.Private.CoreLib.wasm - 200 24.7902 HTTP/2 ***.net
2023-09-07 07:02:37 ::ffff:195.154.***.*** 443 GET /js/LocalStorageAccessor.min.js - 200 2.6798 HTTP/2 ***.net
2023-09-07 07:02:37 ::ffff:195.154.***.*** 443 GET /css/open-iconic/font/fonts/open-iconic.woff - 200 4.9329 HTTP/2 ***.net

With Microsoft Kestrel's Web Server you can use W3CLogging and configure the logging fields you want to enable.

Kestrel is a cross-platform web server that is included and enabled by default in ASP.NET Core

If a field is not available it writes a - to the log file. In my example this happens with cs-uri-query (GoAccess: %q) But it can also happen with, for example, Referrer

In my example I've enabled:

date time c-ip s-port cs-method cs-uri-stem cs-uri-query sc-status time-taken cs-version cs-host

and my GoAccess CLI parameters are:

goaccess *.txt --log-format='%d %t %h %^ %m %U %q %s %L %H %v' --date-format='%Y-%m-%d' --time-format='%H:%M:%S'