Multiple different log formats with a merge report

[LeeNX]

Worst subject ever!! Sorry!

This feature request is linked to #190 and #78 ... I am thinking I would need this kind of thing in future, where using the logs from the web server and the reverse proxy, which both would have two different log formats, merged into one report. Not sure how accurate the representation might be?

Something like goaccess -p1 apache.conf -f1 access.log -p2 varnish.conf -p varnishncsa.log

Thanks

[allinurl]

I think this would have to be done within one config file to avoid conflicts between options. I'm thinking something like:

log-format  %h %^[%d:%^] "%r" %s %b "%R" "%u" access.log
log-format  %h %^[%d:%^] "%r" %s %b "%R" "%u" varnish.log

[LeeNX]

Silly question, would you add the log file in the config file or pass the log file on the command line? Don't think goaccess would be able to match the logs files by corresponding matched patterns?

[felquis]

I'm using cat cat files/*log > combined.log and parsing combined.log into goaccess...

[nvtkaszpir]

how about:

log-format  %h %^[%d:%^] "%r" %s %b "%R" "%u"

[access.log]
log-format  %h %^[%d:%^] "%r" %s %b "%R" "%u"

[varnish.log]
log-format  %h %^[%d:%^] "%r" %s %b "%R" "%u"

This way it is backward compatible with option to adjust log formats per file. Adding log file at the end may be problematic, especially when you process from stdin.

[LeeNX]

Short term fix, changed both the web server and reverse proxy logs to share a format and add extra details per format after "%~" in the goaccessrc file. Works for making goaccess reports but other local log reporting tools not happy, but I can live with that for now.

Hoping that we can get the hit/miss counter #190 / #291 .... '-)

[aadlung]

Hi, is this still "evaluated"? I have the same issue, as I have pages with HTTPS (and reverse proxy for SSL termination), and pages without HTTPS. This results in some log files with the standard %h host variable, but others have the x-forwarded-for, in another field of the log (at the very end)...

[allinurl]

@aadlung it certainly is. I just need to fix some pending issues before I get to this. Thanks for the reminder!

[CraigNoble]

@allinurl been a little over a year now, any update on this?

[allinurl]

@CraigNoble I agree, this release is way behind. I'm addressing the new on-disk storage which will be the cutoff for the next version. I appreciate the reminder, I'll make sure to prioritize this on the following release.

[Pwuts]

bump

[anthonysomerset]

my suggestion would be to add an optional additional parameter to log-file parameter to specify the log-format e.g. log-file /var/log/xferlog ftplog

then additional log formats could be defined like log-format-<name> e.g. log-format-ftplog %h %^[%d:%^] "%r" %s %b "%R" "%u"

if first not provided then you default to the default log-format, if the latter is not provided but is referred to - we should either send some error or fall back to the default

[hirsebrand]

bump

[dmaziuk]

My $.02 would be to split the pipeline into log reader->store and the reporter that works off the store. Looks to me like this is already happening if you use keep-db-file true?

[allinurl]

@dmaziuk yes, it's using a similar model.

[michaelsmoody]

Good afternoon,

We're attempting to parse a number of log files, 4-5 of which have a different format for analytics reasons. As a result of this, it would be ideal to be able to configure goaccess to obey a line such as:

log-file /var/log/httpd/site.example.com-access_log log-format common
log-file /var/log/httpd/site2.example.com-access_log log-format combined

Alternatively, https://github.com/allinurl/goaccess/issues/213#issuecomment-210969069 is also reasonable, but would quickly increase config file size, but I'm happy with either.

For those who are trying to do this, my workaround is currently to setup multiple CustomLog directives in the VirtualHost definition, allowing me to have two formats on the disk. It's also not ideal, as we're doubling the disk space necessary for logs for those sites, but it does work.

I'll watch this issue for any movement on allowing this to be defined in the config file, which would definitely be preferable.

[crbon]

bump

[allinurl]

@crbon Thanks for the reminder on this. As soon as I finish #117, I'll work on this (and several other cool features). It should be sooner than later. Stay tuned..

[michaelsmoody]

Thank you for the update! I'd actually all but assumed this might never get addressed....

I'll look forward to an update in a future release! I'm still using my work around, but it is disk space heavy. Thank you again for your excellent work here.

As a reminder for all of us:

Thus, once again, thank you.