Failed shards in kibana

[VN1977]

Hi! We have a new installation of SELKS6. I upgraded to the last version of all components. While I open kibana I see such an error Of course index logstash-2020..... doesn't have this field flow_id but there is scripted field Correlate Flow to form link to Evebox, so error exists.

[pevma]

Where does this error happen , on which dashboard ?

[VN1977]

It happens on standart discovery page with index pattern logstash-*

[VN1977]

Maybe this code solves the problem

if (doc.containsKey('flow_id'))
  { if (doc['flow_id'].size() != 0)
    {return doc['flow_id'].value}
  }

or

if (doc.containsKey('flow_id'))
  { if (doc['flow_id'].value != null)
    {return doc['flow_id'].value}
  }

[VN1977]

The same trouble with FPC field.

[VN1977]

Replaced scripted field FPC with this code

def src_ip = doc.containsKey('src_ip.keyword') ? (doc['src_ip.keyword'].value != null ? doc['src_ip.keyword'].value : '') : '';
def src_port = doc.containsKey('src_port') ? (doc['src_port'].value != null ? doc['src_port'].value : '') : '';
def dest_ip = doc.containsKey('dest_ip.keyword') ? (doc['dest_ip.keyword'].value != null ? doc['dest_ip.keyword'].value : '') : '';
def dest_port = doc.containsKey('dest_port') ? (doc['dest_port'].value != null ? doc['dest_port'].value : '') : '';
def proto = doc.containsKey('proto.keyword') ? (doc['proto.keyword'].value != null ? doc['proto.keyword'].value : '') : '';
return ('ip.src == ' + src_ip + ' && port.src == ' + src_port + ' && ip.dst == ' + dest_ip + ' && port.dst == ' + dest_port + ' && protocols == ' + proto.toLowerCase());

No errors any more

[pevma]

Thanks for the feedback! I am thinking actually the easiest might be to just replace the whole line with commuity_id - https://github.com/OISF/suricata/blob/master/suricata.yaml.in#L120 and set up Moloch in a similar way the deployments.

[VN1977]

I tried to do so.

[pevma]

Yep - but first you need to enable it in the suricata config, restart suricata and then it will be in the newly generated log.

[VN1977]

I have enabled this option. But I had field communityid in Kibana already before. And now nothing has changed. My mistake in scripted field was in usage of community_id instead of community_id.keyword and after correcting I don't have any errors. But another problem is that no one Comminity ID from Kibana can be found in Moloch.

[pevma]

Think you can fix the error by importing the newest vizs form the API (or directly form the Kibana GUI ) https://github.com/StamusNetworks/KTS7#how-to-use

[collse]

I got a similar one on the SN-All after importing the latest

org.elasticsearch.index.fielddata.ScriptDocValues$Longs.get(ScriptDocValues.java:121) org.elasticsearch.index.fielddata.ScriptDocValues$Longs.getValue(ScriptDocValues.java:115) 'ip == ' + doc['src_ip.keyword'].value + ' && port == ' + doc['src_port'].value + ' && ip == ' + doc['dest_ip.keyword'].value + ' && port == ' + doc['dest_port'].value + ' && protocols == ' + doc['proto.keyword'].value.toLowerCase() ^---- HERE

this script

'ip == ' + doc['src_ip.keyword'].value + ' && port == ' + doc['src_port'].value + ' && ip == ' + doc['dest_ip.keyword'].value + ' && port == ' + doc['dest_port'].value + ' && protocols == ' + doc['proto.keyword'].value.toLowerCase()

Lang painless Position offset 73 Position start 0 Position end 232 Caused by type illegal_state_exception Caused by reason A document doesn't have a value for a field! Use doc[].size()==0 to check if a document is missing a field!

and this on the SN-stats

Type script_exception Reason runtime error Script stack org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:88) org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:40) doc['flow_id'].value ^---- HERE

Script doc['flow_id'].value

Lang painless Position offset 4 Position start 0 Position end 20 Caused by type illegal_argument_exception Caused by reason No field found for [flow_id] in mapping with types []

[pevma]

Can you please paste the output of dpkg -l |grep stamus ?

[collse]

sure

dpkg -l |grep stamus ii kibana-dashboards-stamus 2020042401 amd64 Kibana 6 dashboard templates. ii libhtp2 1:0.5.36-0stamus0 amd64 HTTP normalizer and parser library ii selks-scripts-stamus 2020121401 amd64 SELKS Scritps by StamusN. ii suricata 1:2020121102-0stamus0 amd64 Suricata open source multi-thread IDS/IPS/NSM system.

I downloaded the recent dashboards manually and imported via kibana | https://github.com/StamusNetworks/KTS7 https://codeload.github.com/StamusNetworks/KTS7/zip/master

[pevma]

It seems you don’t have the latest stamus Kibana dashboards pkg. (the one you got is from April (202004...) Can you run the “selks-upgrade_stamus” command procedure for upgrading ? (If this is prod , please try it in QA/test first )

-- Regards, Peter Manev

On 22 Dec 2020, at 16:57, Sev [email protected] wrote:

 sure

dpkg -l |grep stamus ii kibana-dashboards-stamus 2020042401 amd64 Kibana 6 dashboard templates. ii libhtp2 1:0.5.36-0stamus0 amd64 HTTP normalizer and parser library ii selks-scripts-stamus 2020121401 amd64 SELKS Scritps by StamusN. ii suricata 1:2020121102-0stamus0 amd64 Suricata open source multi-thread IDS/IPS/NSM system.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[collse]

I already had upgraded them manually and imported via kibana | https://github.com/StamusNetworks/KTS7 https://codeload.github.com/StamusNetworks/KTS7/zip/master but it seems there were key differences?

had to reset to defaults via the menu to resolve this issue - thanks

[pevma]

You can probably also use the “overwrite import “ option via the rest api too.

-- Regards, Peter Manev

On 22 Dec 2020, at 19:05, Sev [email protected] wrote:

 I already had upgraded them manually and imported via kibana | https://github.com/StamusNetworks/KTS7 https://codeload.github.com/StamusNetworks/KTS7/zip/master but it seems there were key differences?

had to reset to defaults via the menu to resolve this issue - thanks

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.