msiempy icon indicating copy to clipboard operation
msiempy copied to clipboard

Published 20 hours ago • verified publisher icon mfesiem

Rule_NDSNormSigID.msg breaks Queries in 11.3.2

Open lzdx2812 opened this issue 4 years ago • 10 comments

Describe When trying to get Rule_NDSNormSigID.msg, all results break, for example:

Result without asking for Rule_NDSNormSigID.msg field

| LastTime | Rule.msg | | 10/27/2020 14:10:44 | TCP Scan (horizontal) |

Result asking for Rule_NDSNormSigID.msg

| LastTime | Rule.msg | Rule_NDSNormSigID.msg | | 10/27/2020 14:10:44 | Unknown_0 | TCP Scan (horizontal) |

Results are mixed, Rule.msg gents wrong info, Rule_NDSNormSigID.msg contains Rule.msg, Rule_NDSNormSigID.msg shoud contain correct normalization name, however it puts that "unknown_0".

SIEM and msiempy versions:

  • msiempy: 0.3.5
  • ESM version: breaks in 11.3.2, works in 10.4

Additional context Every other query works with every field except Rule_NDSNormSigID.msg

Code Used:

from msiempy import EventManager, FieldFilter print('Simple event query sorted by AlertID') events = EventManager( time_range='PREVIOUS_DAY', fields=['LastTime','Rule.msg','Rule_NDSNormSigID.msg'], filters=[ FieldFilter('SrcIP', ['X.X.X.X'], operator='IN'), ], order=(('ASCENDING', 'AlertID')), limit=100) # Will only load 100 events (per query) events.load_data() print(events) print(events.get_text(fields=['LastTime','Rule.msg','Rule_NDSNormSigID.msg']))

lzdx2812 avatar Oct 28 '20 17:10 lzdx2812

Hello,

Thanks for raising this issue, I'll do my best to fix it.

This is strange... The "Unknown_0" value do not come from the msiempy library.

Can you enable verbose and configure a logfile in your config file ? Then paste (carefully) the logs here please so I can investigate !

Thanks!

tristanlatr avatar Oct 28 '20 17:10 tristanlatr

The more I think about it the more I believe this is not a library issue but more a McAfee issue...

But to be 100% certain of that, please adjust this standalone script to get fields 'LastTime','Rule.msg','Rule_NDSNormSigID.msg' on one side and 'LastTime','Rule.msg' on the other and compare the raw results.

https://github.com/mfesiem/msiem/blob/master/standalone/offsetIssue.py

tristanlatr avatar Oct 28 '20 18:10 tristanlatr

Thank you, Here you will see the debug from de Script I used, asking for Rule_NDSNormSigID.msg

2020-10-28 12:45:20,042 - DEBUG - Calling nitro request : event_query kwargs={'time_range': 'PREVIOUS_DAY', 'order_direction': 'ASCENDING', 'order_field': 'AlertID', 'fields': [{'name': 'Rule.msg'}, {'name': 'LastTime'}, {'name': 'Rule_NDSNormSigID.msg'}, {'name': 'IPSIDAlertID'}], 'filters': [{'type': 'EsmFieldFilter', 'field': {'name': 'SrcIP'}, 'operator': 'IN', 'values': [{'type': 'EsmBasicValue', 'value': '27.156.248.78'}]}], 'limit': 100, 'offset': 0, 'includeTotal': False} 2020-10-28 12:45:20,043 - DEBUG - Calling nitro request : login kwargs=*** 2020-10-28 12:45:20,044 - DEBUG - Requesting HTTP post login *** 2020-10-28 12:45:31,158 - DEBUG - Returning raw requests Response object : <Response [201]> 2020-10-28 12:45:31,160 - DEBUG - Unpacking SIEM response: { "privileges" : { "master" : true, "admin" : false, "power" : false, "audit" : false, "crypto" : false, "systemSettings" : { "read" : true, "write" : true }, 2020-10-28 12:45:31,165 - DEBUG - Calling nitro request : build_stamp kwargs={} 2020-10-28 12:45:31,166 - DEBUG - Requesting HTTP post essmgtGetBuildStamp with data None 2020-10-28 12:45:31,473 - DEBUG - Unpacking SIEM response: { "return": { "buildStamp" : "11.3.2 20200730152039" } } 2020-10-28 12:45:31,479 - DEBUG - <Response [200]> -> Result (<class 'dict'>): {'buildStamp': '11.3.2 20200730152039'} 2020-10-28 12:45:31,482 - DEBUG - Requesting HTTP post v2/qryExecuteDetail?type=EVENT&reverse=false with data {'config': {'timeRange': 'PREVIOUS_DAY', 'fields': [{'name': 'Rule.msg'}, {'name': 'LastTime'}, {'name': 'Rule_NDSNormSigID.msg'}, {'name': 'IPSIDAlertID'}], 'filters': [{'type': 'EsmFieldFilter', 'field': {'name': 'SrcIP'}, 'operator': 'IN', 'values': [{'type': 'EsmBasicValue', 'value': '27.156.248.78'}]}], 'limit': 100, 'offset': 0, 'order': [{'field': {'name': 'AlertID'}, 'direction': 'ASCENDING'}]}} 2020-10-28 12:45:32,629 - DEBUG - Unpacking SIEM response: { "totalRows" : 0, "stopTime" : "10/28/2020 00:00:00", "resultID" : "1224883172", "totalResultID" : "0", "groupByString" : "", "startTime" : "10/27/2020 00:00:00", "countColumn" : 0, " 2020-10-28 12:45:32,635 - DEBUG - <Response [200]> -> Result (<class 'dict'>): {'totalRows': 0, 'stopTime': '10/28/2020 00:00:00', 'resultID': '1224883172', 'totalResultID': '0', 'groupByString': '', 'startTime': '10/27/2020 00:00:00', 'countColumn': 0, 'labelColumn': 0, 'attrib 2020-10-28 12:45:32,638 - DEBUG - Waiting for EsmRunningQuery object : {'totalRows': 0, 'stopTime': '10/28/2020 00:00:00', 'resultID': '1224883172', 'totalResultID': '0', 'groupByString': '', 'startTime': '10/27/2020 00:00:00', 'countColumn': 0, 'labelColumn': 0, 'attributeColumn': 0, 'drilldownColumn': 1} 2020-10-28 12:45:32,641 - DEBUG - Waiting for the query to be executed on the SIEM... 2020-10-28 12:45:32,641 - DEBUG - Calling nitro request : query_status kwargs={'resultID': '1224883172'} 2020-10-28 12:45:32,646 - DEBUG - Requesting HTTP post v2/qryGetStatus with data {'resultID': 1224883172} 2020-10-28 12:45:32,822 - DEBUG - Unpacking SIEM response: { "complete" : false, "percentComplete" : 0, "milliseconds" : 164 } 2020-10-28 12:45:32,826 - DEBUG - <Response [200]> -> Result (<class 'dict'>): {'complete': False, 'percentComplete': 0, 'milliseconds': 164} 2020-10-28 12:45:33,029 - DEBUG - Calling nitro request : query_status kwargs={'resultID': '1224883172'} 2020-10-28 12:45:33,030 - DEBUG - Requesting HTTP post v2/qryGetStatus with data {'resultID': 1224883172} 2020-10-28 12:45:33,160 - DEBUG - Unpacking SIEM response: { "complete" : false, "percentComplete" : 0, "milliseconds" : 537 } 2020-10-28 12:45:33,162 - DEBUG - <Response [200]> -> Result (<class 'dict'>): {'complete': False, 'percentComplete': 0, 'milliseconds': 537} 2020-10-28 12:45:33,364 - DEBUG - Calling nitro request : query_status kwargs={'resultID': '1224883172'} 2020-10-28 12:45:33,365 - DEBUG - Requesting HTTP post v2/qryGetStatus with data {'resultID': 1224883172} 2020-10-28 12:45:33,576 - DEBUG - Unpacking SIEM response: { "complete" : false, "percentComplete" : 99, "milliseconds" : 907 } 2020-10-28 12:45:33,579 - DEBUG - <Response [200]> -> Result (<class 'dict'>): {'complete': False, 'percentComplete': 99, 'milliseconds': 907} 2020-10-28 12:45:33,782 - DEBUG - Calling nitro request : query_status kwargs={'resultID': '1224883172'} 2020-10-28 12:45:33,784 - DEBUG - Requesting HTTP post v2/qryGetStatus with data {'resultID': 1224883172} 2020-10-28 12:45:34,044 - DEBUG - Unpacking SIEM response: { "complete" : true, "percentComplete" : 100, "milliseconds" : 1338 } 2020-10-28 12:45:34,048 - DEBUG - <Response [200]> -> Result (<class 'dict'>): {'complete': True, 'percentComplete': 100, 'milliseconds': 1338} 2020-10-28 12:45:34,049 - DEBUG - Calling nitro request : query_result kwargs={'startPos': 0, 'numRows': 100, 'resultID': '1224883172'} 2020-10-28 12:45:34,052 - DEBUG - Requesting HTTP post v2/qryGetResults?startPos=0&numRows=100&reverse=false with data {'resultID': 1224883172} 2020-10-28 12:45:34,536 - DEBUG - Unpacking SIEM response: { "columns" : [ { "name" : "Rule.msg" }, { "name" : "Alert.LastTime" }, { "name" : "Rule_NDSNormSigID.msg" }, { "name" : "Alert.IPSIDAlertID" } ], "rows" : [ { "values" 2020-10-28 12:45:34,547 - DEBUG - <Response [200]> -> Result (<class 'dict'>): {'columns': [{'name': 'Rule.msg'}, {'name': 'Alert.LastTime'}, {'name': 'Rule_NDSNormSigID.msg'}, {'name': 'Alert.IPSIDAlertID'}], 'rows': [{'values': ['Internet Policy', '10/27/2020 14:10:44', 'Unkno 2020-10-28 12:45:34,552 - DEBUG - Calling nitro request : close_query kwargs={'resultID': '1224883172'} 2020-10-28 12:45:34,555 - DEBUG - Requesting HTTP post v2/qryClose with data {'resultID': 1224883172} 2020-10-28 12:45:34,796 - DEBUG - Unpacking SIEM response: 2020-10-28 12:45:34,799 - DEBUG - <Response [200]> -> Result (<class 'str'>):

lzdx2812 avatar Oct 28 '20 19:10 lzdx2812

Here is the output from the standalone script when Rule_NDSNormSigID.msg is NOT asked for (is OK I think):

POSTING / "" POSTING qryExecuteDetail?type=EVENT&reverse=false {"config": {"timeRange": "PREVIOUS_DAY", "fields": [{"name": "LastTime"}, {"name": "Rule.msg"}, {"name": "SrcIP"}], "filters": [{"type": "EsmFieldFilter", "field": {"name": "SrcIP"}, "operator": "IN", "values": [{"type": "EsmBasicValue", "value": "27.156.248.78"}]}], "limit": 100, "offset": 0}} POSTING qryExecuteDetail?type=EVENT&reverse=false {"config": {"timeRange": "PREVIOUS_DAY", "fields": [{"name": "LastTime"}, {"name": "Rule.msg"}, {"name": "SrcIP"}], "filters": [{"type": "EsmFieldFilter", "field": {"name": "SrcIP"}, "operator": "IN", "values": [{"type": "EsmBasicValue", "value": "27.156.248.78"}]}], "limit": 100, "offset": 50}} POSTING qryGetStatus {"resultID": {"value": 1699380593}} POSTING qryGetStatus {"resultID": {"value": 1721154706}} POSTING qryGetResults?startPos=0&numRows=1&reverse=false {"resultID": {"value": 1699380593}} Result #1 { "return": { "columns" : [ { "name" : "Alert.LastTime" }, { "name" : "Rule.msg" }, { "name" : "Alert.SrcIP" } ], "rows" : [ { "values" : [ "10/27/2020 14:10:08", "FortiGate_UTM Attack Event IPS Custom", "27.156.248.78" ] } ] } } POSTING qryGetResults?startPos=0&numRows=1&reverse=false {"resultID": {"value": 1721154706}} Result #2 { "return": { "columns" : [ { "name" : "Alert.LastTime" }, { "name" : "Rule.msg" }, { "name" : "Alert.SrcIP" } ], "rows" : [ { "values" : [ "10/27/2020 14:10:08", "FortiGate_UTM Attack Event IPS Custom", "27.156.248.78" ] } ] } }

lzdx2812 avatar Oct 28 '20 19:10 lzdx2812

Here is the output from the standalone script when Rule_NDSNormSigID.msg is asked for (Not Okay I think)

POSTING / "" POSTING qryExecuteDetail?type=EVENT&reverse=false {"config": {"timeRange": "PREVIOUS_DAY", "fields": [{"name": "LastTime"}, {"name": "Rule.msg"}, {"name": "Rule_NDSNormSigID.msg"}, {"name": "SrcIP"}], "filters": [{"type": "EsmFieldFilter", "field": {"name": "SrcIP"}, "operator": "IN", "values": [{"type": "EsmBasicValue", "value": "27.156.248.78"}]}], "limit": 100, "offset": 0}} POSTING qryExecuteDetail?type=EVENT&reverse=false {"config": {"timeRange": "PREVIOUS_DAY", "fields": [{"name": "LastTime"}, {"name": "Rule.msg"}, {"name": "Rule_NDSNormSigID.msg"}, {"name": "SrcIP"}], "filters": [{"type": "EsmFieldFilter", "field": {"name": "SrcIP"}, "operator": "IN", "values": [{"type": "EsmBasicValue", "value": "27.156.248.78"}]}], "limit": 100, "offset": 50}} POSTING qryGetStatus {"resultID": {"value": 2514887835}} POSTING qryGetStatus {"resultID": {"value": 1483268491}} POSTING qryGetResults?startPos=0&numRows=1&reverse=false {"resultID": {"value": 2514887835}} Result #1 { "return": { "columns" : [ { "name" : "Alert.LastTime" }, { "name" : "Rule.msg" }, { "name" : "Rule_NDSNormSigID.msg" }, { "name" : "Alert.SrcIP" } ], "rows" : [ { "values" : [ "10/27/2020 14:10:08", "Exploit", "Unknown_0", "" ] } ] } } POSTING qryGetResults?startPos=0&numRows=1&reverse=false {"resultID": {"value": 1483268491}} Result #2 { "return": { "columns" : [ { "name" : "Alert.LastTime" }, { "name" : "Rule.msg" }, { "name" : "Rule_NDSNormSigID.msg" }, { "name" : "Alert.SrcIP" } ], "rows" : [ { "values" : [ "10/27/2020 14:10:08", "Exploit", "Unknown_0", "" ] } ] } }

lzdx2812 avatar Oct 28 '20 19:10 lzdx2812

I'm now formal, you have found a bug in the McAfee SIEM ! Now you can open a support ticket to get this fixed :D Congratulation @lzdx2812 !

As a reference this, type of output is the issue. It mixes up everything and totally remove one of the results fields, for you, that was ScrIP, in my case it's IPSIDAlertID.

DEBUG - Unpacking SIEM response: {
  "columns" : [ {
    "name" : "Rule_NDSNormSigID.msg"
  }, {
    "name" : "Rule.msg"
  }, {
    "name" : "Alert.IPSIDAlertID"
  }, {
    "name" : "Alert.LastTime"
  } ],
  "rows" : [ {
    "values" : [ "User Logon", "Unknown_0", "10/28/2020 19:37:52.000", "" ]
  }, {
    "values" : [ "User Logon", "Unknown_0", "10/28/2020 19:37:33.000", "" ]
  }, {
    "values" : [ "User Logon", "Unknown_0", "10/28/2020 19:36:58.000", "" ]
  }, {
    "values" : [ "User Logon", "Unknown_0", "10/28/2020 19:36:32.000", "" ]
  }, {
    "values" : [ "User Logon", "Unknown_0", "10/28/2020 19:35:50.000", "" ]
  }, {
    "values" : [ "Postfix Connect from host", "Unknown_0", "10/28/2020 19:34:18.000", "" ]
  }, {
    "values" : [ "Postfix Disconnect from host", "Unknown_0", "10/28/2020 19:34:18.000", "" ]
  }, {
    "values" : [ "Postfix Disconnect from host", "Unknown_0", "10/28/2020 19:34:18.000", "" ]
  }, {
    "values" : [ "Postfix Client message transfer", "Unknown_0", "10/28/2020 19:34:18.000", "" ]
  }, {
    "values" : [ "Postfix Message moved to active queue for delivery", "Unknown_0", "10/28/2020 19:34:18.000", "" ]
  } ]
}

The same query, without Rule_NDSNormSigID.msg field give a normal results:

DEBUG - Unpacking SIEM response: {
  "columns" : [ {
    "name" : "Alert.LastTime"
  }, {
    "name" : "Alert.IPSIDAlertID"
  }, {
    "name" : "Rule.msg"
  } ],
  "rows" : [ {
    "values" : [ "10/28/2020 19:43:13", "144115188075855872|18477", "User Logon" ]
  }, {
    "values" : [ "10/28/2020 19:42:13", "144115188075855872|18476", "User Logon" ]
  }, {
    "values" : [ "10/28/2020 19:41:22", "144115188075855872|18475", "User Logon" ]
  }, {
    "values" : [ "10/28/2020 19:37:52", "144115188075855872|18474", "User Logon" ]
  }, {
    "values" : [ "10/28/2020 19:37:33", "144115188075855872|18473", "User Logon" ]
  }, {
    "values" : [ "10/28/2020 19:36:58", "144115188075855872|18472", "User Logon" ]
  }, {
    "values" : [ "10/28/2020 19:36:32", "144115188075855872|18471", "User Logon" ]
  }, {
    "values" : [ "10/28/2020 19:35:50", "144115188075855872|18470", "User Logon" ]
  }, {
    "values" : [ "10/28/2020 19:34:18", "144116287604260864|646376", "Postfix Connect from host" ]
  }, {
    "values" : [ "10/28/2020 19:34:18", "144116287604260864|646378", "Postfix Disconnect from host" ]
  } ]
}

I'm using SIEM 11.3.0

tristanlatr avatar Oct 28 '20 19:10 tristanlatr

I'll keep this issue open. Please keep us posted about the status of your support ticket ! :)

tristanlatr avatar Oct 28 '20 20:10 tristanlatr

will do, thank you!

lzdx2812 avatar Oct 28 '20 23:10 lzdx2812

Any news @lzdx2812 ?

Thanks

tristanlatr avatar Jul 10 '21 03:07 tristanlatr

Hi!, support was not helpful at all about this, they simply acknowledge the error but did no more.

lzdx2812 avatar Jul 11 '21 19:07 lzdx2812