output-json: Enrich eve-log with geoip information from libmaxminddb (target master)

[fandigunawan]

I faced a very large log to be enriched using geoip. At the moment I utilize logstash to enrich the log file from Suricata but at the cost of additional resources in RAM, CPU, and storage. I would like to optimize this by adding optional geoip enrichment into Suricata.

This PR is replacement of https://github.com/OISF/suricata/pull/10565 to target master branch.

• [ X ] I have read the contributing guide lines at https://docs.suricata.io/en/latest/devguide/contributing/contribution-process.html
• [ X ] I have signed the Open Information Security Foundation contribution agreement at https://suricata.io/about/contribution-agreement/ (note: this is only required once)
• [ X ] I have updated the user guide (in doc/userguide/) to reflect the changes made (if applicable)

Link to redmine ticket: 6999

Describe changes:

• I adds optional geoip enrichment into Eve log by setting geoip-enrichment option under eve-log configuration in suricata.yaml
• The JSON structure of geoip is based on Elastic ECS geo specification

Eve output example with enrichment

{
    "timestamp": "2021-05-27T03:37:44.575843+0700",
    "flow_id": 2130735805455113,
    "pcap_cnt": 15756,
    "event_type": "fileinfo",
    "geoip_src": {
        "ip": "192.236.155.230",
        "geo": {
            "continent_code": "NA",
            "country_iso_code": "US",
            "city_name": "Seattle",
            "country_name": "United States",
            "continent_name": "North America",
            "timezone": "America/Los_Angeles",
            "location": {
                "lat": 47.4902,
                "lon": -122.3004
            }
        }
    },
    "geoip_dst": {},
    "src_ip": "192.236.155.230",
    "src_port": 80,
    "dest_ip": "10.5.26.4",
    "dest_port": 56042,
    "proto": "TCP",
    "pkt_src": "wire/pcap",
    "http": {
        "hostname": "192.236.155.230",
        "url": "/images/redbutton.png",
        "http_user_agent": "WinHTTP loader/1.0",
        "http_content_type": "Content-type: application/octet-stream",
        "http_method": "GET",
        "protocol": "HTTP/1.1",
        "status": 200,
        "length": 105556
    },
    "app_proto": "http",
    "fileinfo": {
        "filename": "/images/redbutton.png",
        "gaps": false,
        "state": "TRUNCATED",
        "stored": false,
        "size": 102400,
        "tx_id": 0
    },
}

[github-actions[bot]]

NOTE: This PR may contain new authors.

[github-actions[bot]]

NOTE: This PR may contain new authors.

[catenacyber]

Could you run ./scripts/clang-format.sh rewrite-branch to fix the CI formatting check ?

[fandigunawan]

Could you run ./scripts/clang-format.sh rewrite-branch to fix the CI formatting check ?

I have run clang-format on the branch

[github-actions[bot]]

NOTE: This PR may contain new authors.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 73.33333% with 4 lines in your changes are missing coverage. Please review.

Project coverage is 83.00%. Comparing base (7d937db) to head (0089c5e). Report is 245 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10703      +/-   ##
==========================================
+ Coverage   82.69%   83.00%   +0.30%     
==========================================
  Files         926      917       -9     
  Lines      247637   248684    +1047     
==========================================
+ Hits       204790   206424    +1634     
+ Misses      42847    42260     -587     

Flag	Coverage Δ
fuzzcorpus	64.25% <53.33%> (+0.20%)	:arrow_up:
suricata-verify	62.69% <73.33%> (+0.71%)	:arrow_up:
unittests	62.32% <0.00%> (+0.14%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

[catenacyber]

Link to redmine ticket: none

Thanks for the work.

I do not know about the content of this PR, but I know that it will need a redmine ticket if we want it

[jufajardini]

At the moment I utilize logstash to enrich the log file from Suricata but at the cost of additional resources in RAM, CPU, and storage. I would like to optimize this by adding optional geoip enrichment into Suricata.

Ticket: https://redmine.openinfosecfoundation.org/issues/6999 @fandigunawan could you please create a redmine user and ask us to assign this ticket to you, so we can keep proper track of this work - and register feedback and decisions? :)

[fandigunawan]

At the moment I utilize logstash to enrich the log file from Suricata but at the cost of additional resources in RAM, CPU, and storage. I would like to optimize this by adding optional geoip enrichment into Suricata.

Ticket: https://redmine.openinfosecfoundation.org/issues/6999 @fandigunawan could you please create a redmine user and ask us to assign this ticket to you, so we can keep proper track of this work - and register feedback and decisions? :)

Hi, I have created Redmine user here https://redmine.openinfosecfoundation.org/users/10064

[jufajardini]

At the moment I utilize logstash to enrich the log file from Suricata but at the cost of additional resources in RAM, CPU, and storage. I would like to optimize this by adding optional geoip enrichment into Suricata.

Ticket: https://redmine.openinfosecfoundation.org/issues/6999 @fandigunawan could you please create a redmine user and ask us to assign this ticket to you, so we can keep proper track of this work - and register feedback and decisions? :)

Hi, I have created Redmine user here https://redmine.openinfosecfoundation.org/users/10064

Thanks, assigned it to you :)

[jufajardini]

Updated description to include Redmine ticket. If we do accept these changes, the ticket number shall also be included as part of the commit messages (example: https://github.com/OISF/suricata/commit/3643b6ed4bed480f3d2608e38407d58709fdc3d8)

[fandigunawan]

Updated description to include Redmine ticket. If we do accept these changes, the ticket number shall also be included as part of the commit messages (example: 3643b6e)

Hi, I have updated the commit message using Redmine ticket number 6999

[jufajardini]

Updated description to include Redmine ticket. If we do accept these changes, the ticket number shall also be included as part of the commit messages (example: 3643b6e)

Hi, I have updated the commit message using Redmine ticket number 6999

Thanks! Could you please have a look at the ticket? Some considerations were left there by Jason, from the team :)

[github-actions[bot]]

NOTE: This PR may contain new authors.

[jufajardini]

On another note: if/when there is more feedback to be incorporated, please open a new PR :)

Another thing that we'll ask, before we can merge the final work, is that the commit message follow our guidelines, especially with regards to character limit for the subject line (max should be 50 char). :)

[jasonish]

"geoip_src": { "ip": "192.236.155.230", "geo": { "continent_code": "NA", "country_iso_code": "US", "city_name": "Seattle", "country_name": "United States", "continent_name": "North America", "timezone": "America/Los_Angeles", "location": { "lat": 47.4902, "lon": -122.3004 } } },

Can we change this to be:

    "geoip_src": {
            "continent_code": "NA",
            "country_iso_code": "US",
            "city_name": "Seattle",
            "country_name": "United States",
            "continent_name": "North America",
            "timezone": "America/Los_Angeles",
            "location": {
                "lat": 47.4902,
                "lon": -122.3004
            }
    },

We already have the IP address, and I think the nested geo object doesn't provide a valuable abstraction. Its still close to ECS. Thanks.

[jufajardini]

Hello @fandigunawan did you get a chance to check the feedback left? :)

[victorjulien]

PR is getting pretty stale, I suggest we close it.

[jufajardini]

We're closing this PR as stale according to our PR workflow policies.

We do appreciate your interest and time in contributing to Suricata, so if you would be able to do that again in the future, you'll be most welcome! :)