dashborad error (some chart)

[sysmoon]

hello! my ELK env is es/kibana: 5.4.0

after run load.sh when i opened dashboard all chart using ***.raw field as a aggregator is not visible (no results found). index pattern is ok. what wrong?

[pevma]

If this is a fresh/test installation you can try the following -

find /opt/selks/kibana4-dashboards/ -type f -exec sed -i -e 's/\.raw/\.keyword/g' {} \;

then reload the dashboards. Please let us know how it goes.

[sysmoon]

@hello pevma, i don't have path like that '/opt/selks/kibana4-dashboards/' do i miss something? i just clone KTS5 master branch and run load.sh and my elk version is all 5.4.0.

my log flow is filebeat(read eve.json) -> logstash -> es

[logstash.conf]

input{

  beats {
    port => 5044
    codec => json
  }
}

filter {

  if [type] == "SELKS" {

    date {
      match => [ "timestamp", "ISO8601" ]
    }

    ruby {
      code => "
        if event.get('[event_type]') == 'fileinfo'
          event.set('[fileinfo][type]', event.get('[fileinfo][magic]').to_s.split(',')[0])
        end
      "
    }

    ruby {
      code => "
        if event.get('[event_type]') == 'alert'
          sp = event.get('[alert][signature]').to_s.split(' group ')
          if (sp.length == 2) and /\A\d+\z/.match(sp[1])
            event.set('[alert][signature]', sp[0])
          end
        end
      "
    }

    metrics {
      meter => [ "eve_insert" ]
      add_tag => "metric"
      flush_interval => 30
    }
  }


  if [http] {
    useragent {
      source => "[http][http_user_agent]"
      target => "[http][user_agent]"
    }
  }


  if [src_ip]  {
    geoip {
      source => "src_ip"
      database => "/etc/logstash/db/GeoIP2-City.mmdb"
      target => "geoip"
      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
    }

    mutate {
      convert => [ "[geoip][coordinates]", "float" ]
    }

    if ![geoip.ip] {
      if [dest_ip]  {
        geoip {
          source => "dest_ip"
          target => "geoip"
          database => "/etc/logstash/db/GeoIP2-City.mmdb"
          add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
          add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
        }

        mutate {
          convert => [ "[geoip][coordinates]", "float" ]
        }
      }
    }
  }

  if [dest_ip] {
    if [dest_ip] !~ ":" {
      mutate {
        add_field => [ "[dest_ip4]", "%{dest_ip}" ]
      }
    }
  }
}


output {
  elasticsearch {
    hosts => "localhost"
  }

  if [event_type] and [event_type] not in ['stats', 'engine'] {
    elasticsearch {
      hosts => "localhost"
      index => "logstash-%{event_type}-%{+YYYY.MM.dd}"
    }
  } else {
    elasticsearch {
      hosts => "localhost"
      index => "logstash-%{+YYYY.MM.dd}"
    }
  }
}

[sysmoon]

solve like this. "find ***/KTS5/dashboards/ -type f -exec sed -i -e 's/.raw/.keyword/g' {} ;" and dashboard working fine. but some chart still not visible still. i will find reason ^^

thank you pevma.

[pevma]

Ok - please keep us posted. Thank you for digging in. Which charts are problematic?

[sysmoon]

hello pevma.

i summarize the chart with errors.

[SN ALERTS]

• SN Alert-BySmtpHello reason: query error (exist:smtp*)

[SN FILE-Transactions]

• SN FILE-GeoIPPDFAndExecutables reason: geoip.location field is not analyzed

[SN HTTP]

• SN HTTP-AcceptEncoding (No results displayed because all values equal 0.)
• SN HTTP-AcceptEncodingByConnection (No results displayed because all values equal 0.)
• SN HTTP-CacheControl (No results displayed because all values equal 0.) reason: there are no index data (http.accept_encoding.keyyword, http.accept_encoding.keyword, http.connection.keyword, http.cache_control.keyword)

[SN VLAN] all chart is not visible (no results found). vlan field is not indexed. as you mentioned i enabled "use-for-tracking: true" in suricata.yml

except chart i wrote above, almost chart working well.. i wanna you to check chart with errors and my logstash.conf. i think logstash.conf can be a hole..

[pevma]

@sysmoon - Thanks for the feedback! I think only the first one is a bug. The lack of data on the other 3 dashboards has to do with no such data being available. Aka if you don't have VLAN tagged traffic - you should not see anything in the VLAN dashboard.