Any plans for Windows EVTX?

[geekscrapy]

Wondered if there were plans to read EVTX logs?

These are binary xml logs used by Windows. The logs themselves can be converted to XML (and then to JSON for example), however there is heavy use of nested structures and the fields are only standard across events of a similar type (common event ID).

There are very few tools in Linux that allow analysis/reading of these types of logs. The common go to library is: https://github.com/williballenthin/python-evtx This outputs to logs to an XML format which can then be converted to JSON for example. Downsides to this library are that it is pretty slow...!

Would love to see the ability to analyse evtx logs with lnav! Even if there is a manual preprocessing phase.

TIA!

[tstack]

There's this C library: https://github.com/libyal/libevtx/

I'm not sure how well it works, but I can take a look to see if it can be incorporated into lnav.

[tstack]

What do the JSON version of the logs look like? You should be able to use lnav to process the JSON form until native support is added.

[geekscrapy]

You maybe able to write a lnav JSON parser, however, you would have to write one for each individual event ID, of which there are thousands... This is due to the fact that each event ID has a different sub structure.

The following is taken from: https://rawsec.lu/blog/posts/2018/Feb/04/go-evtx-signature-engine/ And it describes the above issue nicely (in the bottom paragraph).

As within the EVTX files the events are stored in BinXML format, it is quite common to represent Windows event in XML format as shown below.

In XML format (once converted, probably using the library you stated above):

<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'>
  <EventData>
    <Data Name='UtcTime'>2018-02-05 18:13:31.315</Data>
    <Data Name='ProcessGuid'>{49F1AF32-1053-5A78-0000-00109473DD01}</Data>
    <Data Name='ProcessId'>2608</Data>
    <Data Name='Image'>C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe</Data>
    <Data Name='ImageLoaded'>\\VBOXSVR\golang-win32\src\win32\wevtapi\test\test.test.exe</Data>
    <Data Name='FileVersion'>?</Data>
    <Data Name='Description'>?</Data>
    <Data Name='Product'>?</Data>
    <Data Name='Company'>?</Data>
    <Data Name='Hashes'>SHA1=62E6250F800ADE743C98B342F4C905C8E64B4A4A,MD5=8E730B5B358DCE3F9F2E773D87BA50F0,SHA256=BA54DDEDFFE1178CA9AD367C286D753A17FD911DC52ED644F73EF0237FC55F84,IMPHASH=2C53CF70BB7ACD75FD60D941F68E3B77</Data>
    <Data Name='Signed'>false</Data>
    <Data Name='Signature'></Data>
    <Data Name='SignatureStatus'>Unavailable</Data>
  </EventData>
  <System>
    <Provider Name='Microsoft-Windows-Sysmon' Guid='{5770385F-C22A-43E0-BF4C-06F5698FFBD9}'/>
    <EventID>7</EventID>
    <Version>3</Version>
    <Level>4</Level>
    <Task>7</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8000000000000000</Keywords>
    <TimeCreated SystemTime='2018-02-05T18:13:31.511688000Z'/>
    <EventRecordID>13185699</EventRecordID>
    <Correlation/>
    <Execution ProcessID='1404' ThreadID='1872'/>
    <Channel>Microsoft-Windows-Sysmon/Operational</Channel>
    <Computer>GenEric-PC</Computer>
    <Security UserID='S-1-5-18'/>
  </System>
</Event>

Under the XML root <Event>, we notice two nodes, which are <EventData> and <System>. The <System> node contains global information about the event and can be seen as a kind of metadata of the event. For instance under this node we can find information such as the <Channel> identifying the source of the event and the <EventID> characterizes the type of the event. The couple formed by the <Channel> and the <EventID> uniquely identifies a type of Windows event. For instance the event above identifies a Sysmon ImageLoad event. One can also find other useful information like the time at which the event has been created in the Windows event logging system, most of the time slightly different from the time at which the event actually occurred.

The <EventData> node contains information specific to the kind of event so any type of event has its own <EventData> definition. Taking the above example as reference, any other Sysmon ImageLoad event will have exactly the same <Data> nodes but of course containing different values. Likewise, a different Windows event like the well known Security Successfull Logon (EventID: 4624) would have a completely different <EventData> definition while the <System> section shape would be the same.

While the XML format is human readable one could prefer using JSON object for better interoperability. Since there is no one to one translation between XML and JSON, we propose the following translation into JSON for the previously shown XML event.

In JSON

{
  "Event":{
    "EventData":{
      "UtcTime":"2018-02-05 18:13:31.315",
      "ProcessGuid":"{49F1AF32-1053-5A78-0000-00109473DD01}",
      "ProcessId":"2608",
      "Image":"C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe",
      "ImageLoaded":"\\\\VBOXSVR\\golang-win32\\src\\win32\\wevtapi\\test\\test.test.exe",
      "FileVersion":"?",
      "Description":"?",
      "Product":"?",
      "Company":"?",
      "Hashes":"SHA1=62E6250F800ADE743C98B342F4C905C8E64B4A4A,MD5=8E730B5B358DCE3F9F2E773D87BA50F0,SHA256=BA54DDEDFFE1178CA9AD367C286D753A17FD911DC52ED644F73EF0237FC55F84,IMPHASH=2C53CF70BB7ACD75FD60D941F68E3B77",
      "Signed":"false",
      "Signature":"",
      "SignatureStatus":"Unavailable"
    },
    "System":{
      "Provider":{
        "Name":"Microsoft-Windows-Sysmon",
        "Guid":"{5770385F-C22A-43E0-BF4C-06F5698FFBD9}"
      },
      "EventID":"7",
      "Version":"3",
      "Level":"4",
      "Task":"7",
      "Opcode":"0",
      "Keywords":"0x8000000000000000",
      "TimeCreated":{
        "SystemTime":"2018-02-05T18:13:31.511688000Z"
      },
      "EventRecordID":"13185699",
      "Correlation":{},
      "Execution":{
        "ProcessID":"1404",
        "ThreadID":"1872"},
        "Channel":"Microsoft-Windows-Sysmon/Operational",
        "Computer":"GenEric-PC",
        "Security":{
          "UserID":"S-1-5-18"
        }
      }
    }
  }

[geekscrapy]

Something I forgot to note above: It maybe best to have lnav process the evtx end to end. This is due to the variety of ways JSON could be generated from the EVTX file. A couple of different converters at the moment:

• Get-WinEvent -LogName system -MaxEvents 1 | convertto-json from: https://hazzy.techanarchy.net/winadmin/windows/windows-powershell-elk-log-wash/
• Using python-evtx: https://gist.github.com/truekonrads/f04ff0409622876d5e6912d78e9f2c5a

[geekscrapy]

Here is a sample System.evtx file you could do testing against: https://github.com/JPCERTCC/LogonTracer/blob/master/sample/Security.evtx

[tstack]

How would you want something like this even rendered on the screen? Looking at the XML you gave, I don't really see a plaintext message in there.

[geekscrapy]

So, when evtx is loaded on a Windows machine symbols are used to provide a "proper" message. For the example given, it would actually be "Image loaded". However, as I say before, there are thousands of these so it's not feasible to have a db of translations.

For reference, this is an explanation of this particular event: https://www.ultimatewindowssecurity.com/securitylog/encyclopedia/event.aspx?eventid=90007 . We use that website (I'm in the security field) to look up event IDs that we dont know.

In terms of how to represent, I'd be looking for something like the following fields to be displayed by default: TimeCreated, Provider Name, Event ID, Computer. With the trailing space filled with the details from the EventData sub object. The extra fields would still be needed to be able to be utilised in SQL query statements. Either as: eventdata.image or directly, image

If you potentially take a look at how LogParser does it, that's probably a good way to go. This is however a Windows only tool 😞 This is a short video showing it in use: https://youtu.be/mCfkFO0xs34

I'll have a chat with the guys at my office to see what fields would be best by default

[FLeven]

Hi, any progress with parsing evtx files ? Since WSL2, lnav could be an interesting alternative to the log parser.

[mplattner]

I'd also be very interested in .evtx support. Are there any news on this issue?