logstash-codec-nmap: Failed to install template

[ffichter]

Hello all,

I followed these steps, trying to index nmap scans into elasticsearch: https://qbox.io/blog/how-to-index-nmap-port-scan-results-into-elasticsearch

But I cannot install the template.

Please advise.

Thanks and regards,

Fred

• Version: logstash 6.0.0 logstash-codec-nmap (0.0.21)

• Operating System: Linux t440s 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u5 (2017-09-19) x86_64 GNU/Linux

• Config File (if you have sensitive info, please remove it): root@t440s:/usr/share/logstash# cat /home/fred/nmap/nmap3-logstash.conf input { file { path => "/home/fred/nmap/*.xml" start_position => "beginning" sincedb_path => "/dev/null" codec => nmap tags => [nmap] } } filter { if "nmap" in [tags] {

  Don't emit documents for 'down' hosts

  if [status][state] == "down" { drop {} } mutate { # Drop HTTP headers and logstash server hostname remove_field => ["headers", "hostname"] } if "nmap_traceroute_link" == [type] { geoip { source => "[to][address]" target => "[to][geoip]" } geoip { source => "[from][address]" target => "[from][geoip]" } } if [ipv4] { geoip { source => ipv4 target => geoip } } } } output { if "nmap" in [tags] { elasticsearch { hosts => "127.0.0.1:9600" document_type => "nmap-reports" document_id => "%{[id]}" # Nmap data usually isn't too bad, so monthly rotation should be fine index => "nmap-logstash-%{+YYYY.MM}" template => "/home/fred/nmap/elasticsearch_nmap_template.json" template_name => "logstash_nmap" } stdout { codec => json_lines } } }

• Sample Data:

• Steps to Reproduce: https://qbox.io/blog/how-to-index-nmap-port-scan-results-into-elasticsearch

• Logs when starting logstash:

[2017-11-15T14:57:53,501][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/home/fred/nmap/elasticsearch_nmap_template.json"} [2017-11-15T14:57:53,556][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"undefined method split' for nil:NilClass", :class=>"NoMethodError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:18:in block in get_es_major_version'", "org/jruby/RubyArray.java:2486:in map'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:18:in get_es_major_version'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:7:in install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.0-java/lib/logstash/outputs/elasticsearch/common.rb:52:in install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.0-java/lib/logstash/outputs/elasticsearch/common.rb:25:in register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:9:in register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:43:in register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:388:in register_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:399:in block in register_plugins'", "org/jruby/RubyArray.java:1734:in each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:399:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:800:in maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:409:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:333:in run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:293:in `block in start'"]}

[praseodym]

I suspect that the problem here is that mapping types have been deprecated in Elasticsearch 6.0, and the template provided by the nmap codec is dependant on them. I'd suggest adapting the pipeline and creating a new template which avoids mapping types.

[bluefangs]

@ffichter - Did you ever happen to prepare the template?

[buzzdeee]

the change in #33 makes it work for /me, can use the template in recent 7.13.x version of the stack