Switch to using logstash geoip filter for all lookups

[bndabbs]

We should centralize all geoip enrichment to be done in Logstash.

• [ ] Remove Bro scripts for geoip
• [ ] Remove MaxMind configuration tasks
• [ ] Add task to install Logstash geoip filter
• [ ] Add geoip filter statement for Bro logs
• [ ] Add geoip filter statement for Suricata logs
• [ ] Update ES mappings to include any new fields

[bndabbs]

This should work for the Bro logs. I have previously tested everything but the ASN lookup.

      cidr {
        address => [ "%{[id_orig_h]}" ]
        network => [ "0.0.0.0/32", "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16", "fc00::/7", "127.0.0.0/8", "::1/128","169.254.0.0/16", "fe80::/10","224.0.0.0/4", "ff00::/8","255.255.255.255/32" ]
        add_field => { "[@meta][orig_host_routable]" => "false" }
      }
      cidr {
        address => [ "%{[id_resp_h]}" ]
        network => [ "0.0.0.0/32", "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16", "fc00::/7", "127.0.0.0/8", "::1/128","169.254.0.0/16", "fe80::/10","224.0.0.0/4", "ff00::/8","255.255.255.255/32" ]
        add_field => { "[@meta][resp_host_routable]" => "false" }
      }

      if ![@meta][orig_host_routable] {
        mutate {
          add_field => {
            "[@meta][orig_host_routable]" => "true"
          }
        }
      }
      if ![@meta][resp_host_routable] {
        mutate {
          add_field => {
            "[@meta][resp_host_routable]" => "true"
          }
        }
      }

      if [@meta][orig_host_routable] == "true" {
        geoip {
          source => "id_orig_h"
          target => "[@meta][geoip_orig]"
          default_database_type => "City"
        }
        geoip {
          source => "id_orig_h"
          target => "[@meta][geoip_orig]"
          default_database_type => "ASN"
        }
      }
      if [@meta][resp_host_routable] == "true" {
        geoip {
          source => "id_resp_h"
          target => "[@meta][geoip_resp]"
          default_database_type => "City"
        }
        geoip {
          source => "id_resp_h"
          target => "[@meta][geoip_resp]"
          default_database_type => "ASN"
        }
      }

[lrdfang]

I can confirm the code works for everything to include ASN. I had to tweak the field names a little for my application but otherwise it is good.

[neu5ron]

lets use a variation of my scripts for HELK project that will cleanup IPs. Also, this allows preventing lookups on IP addresses that would never have an ASN or City/Geo such as (127.0.0.1, 239.255.1.1, etc reserved RFCs like RFC5736, RFC1122-3.2.1.3, RFC2544, RFC5737. Its also much faster than CIDR, because its using integer conversions with even only checking the beginning numbers of necessary.

this first one cleans up https://github.com/Cyb3rWard0g/HELK/blob/master/docker/helk-logstash/pipeline/8012-dst-ip-cleanups-filter.conf

then the second one will add city and asn https://github.com/Cyb3rWard0g/HELK/blob/master/docker/helk-logstash/pipeline/8112-dst-ip-filter.conf

Just will obviously need to change fields to corresponding RockNSM field names for Elastic.