startbootstrap icon indicating copy to clipboard operation
startbootstrap copied to clipboard

Allow custom log formats for apache_access parser

Open Towelie opened this issue 3 years ago • 4 comments

Description of problem: Currently the apache_access parser reads just the last ip in the string and does not pick up others from what I have noticed Would it be possible to extend support for custom apache log formats? Specifically I am trying to read logs in this format: LogFormat "%{X-Forwarded-For}i %h %l %u %t "%r" %>s %b %D Example: 1.2.3.4 10.0.0.1 - - [25/Oct/2021:09:01:28 +0000] "GET / HTTP/1.1" 302 280 306 1.2.3.4,192.168.1.1 10.0.0.1 - - [25/Oct/2021:09:01:28 +0000] "GET / HTTP/1.1" 302 280 306 (where the last ip is the internal ip of the host)

Command line and arguments: log2timeline /path/to/file.plaso /path/to/collection

Source data: [Please provide the source data you used when you experienced the problem. For publicly available data please provide an URL or path of the source data. https://github.com/log2timeline/plaso/blob/main/plaso/parsers/apache_access.py

Plaso version: 20211024 Operating system Plaso is running on: RHEL 7.2

Installation method: docker pull log2timeline/plaso If multiple methods were used please indicate.

Debug output/tracebacks:

Please run the tool with "-d" to generate debug output, and include anything relevant. Also see: [Producing debug logs][https://plaso.readthedocs.io/en/latest/sources/Troubleshooting.html#producing-debug-logs]

For more information see the [troubleshooting guide][https://plaso.readthedocs.io/en/latest/sources/Troubleshooting.html]

Towelie avatar Dec 09 '21 14:12 Towelie

@Towelie thx for the report what could help fast track this is to have test data in the formats you would like to see supported.

joachimmetz avatar Dec 09 '21 23:12 joachimmetz

Also any additional context on other Apache related formats could be useful?

joachimmetz avatar Dec 09 '21 23:12 joachimmetz

Also any additional context on other Apache related formats could be useful?

Some middle-tier solutions use implementations of the apache log format, with slightly customized fields, similar to x-forwarded ip if behind a WAF.

Is there any way for you to share a brief how-to on how to make a custom apache_access parser if the desired fields are a bit different? I could try doing that perhaps?

Thank you

Towelie avatar Dec 13 '21 08:12 Towelie

@Towelie examples of the format could help

  1. if these are static/predefined similar log formats we could consider adding them to the existing parser
  2. if these are custom based on a header we could do something similar as the iis parser

I could try doing that perhaps?

Yes please, to get started https://plaso.readthedocs.io/en/latest/sources/developer/index.html

Feel free to send a PR or propose an approach as a comment on this issue.

joachimmetz avatar Dec 13 '21 13:12 joachimmetz