Nmap Output Kibana Ingest - Multiple Values

[spacecwoboy]

I am wanting to run some visualizations/dashboards on these items (like in the demos), but cannot uniquely assign each their own values, and then drill/report them via Kibana.

Other items do expand out via the different nmap event types (they are not all in the same event), such as the breakout of port service info:

first event

later in the pipeline

Nmap command: nmap -sV -O x.x.x.x -oX | curl http://x.x.x.x --data-binary @- Starting Nmap 7.01 ( https://nmap.org ) at 2017-04-05 15:09 CDT

(I found that nmap -A generated nothing but the logstash nmap metadata events)

Architecture: Nmap -> HTTP LS Input -> ES -> Kibana

LS output conf:

if "nmap" in [tags] {
   elasticsearch {
     hosts => ["x","y"]
     index => "nmap-logstash-%{+YYYY.MM.dd}"
     document_type => "%{[type]}"
     document_id => "%{[id]}"
     template => "/etc/logstash/templates/elasticsearch_nmap_template.json"
     template_name => "logstash_nmap"
   }

Kibana does not seem to sift out the variables individually, but they are stacked in some kinda json/array.

Any way to fix this? Is it the template that needs tweaking (one provided in the master-repo)?

Does my nmap command need fixing, or is it just a non-standard output?

Scanning my raspberry pi with this command, and the output in non-xml format is as follows: ` _Starting Nmap 7.01 ( https://nmap.org ) at 2017-04-05 15:27 CDT Nmap scan report for 10.8.1.5 Host is up (0.011s latency). Not shown: 999 closed ports PORT STATE SERVICE VERSION 22/tcp open ssh OpenSSH 6.0p1 Debian 4+deb7u6 (protocol 2.0) MAC Address: X:X:6F (Raspberry Pi Foundation) Device type: general purpose Running: Linux 3.X|4.X OS CPE: cpe:/o:linux:linux_kernel:3 cpe:/o:linux:linux_kernel:4 OS details: Linux 3.2 - 4.0 Network Distance: 1 hop Service Info: OS: Linux; CPE: cpe:/o:linux:linux_kernel

OS and Service detection performed. Please report any incorrect results at https://nmap.org/submit/ . Nmap done: 1 IP address (1 host up) scanned in 4.41 seconds `_

[andrewvc]

@spacecwoboy Thanks for the excellent report!

I don't have time to look in-depth this week, but hopefully next. I can say that this is probably an issue with the template.

Any chance you could spend some time fiddling with the template @spacecwoboy ? Logstash looks to be structuring things correctly, but perhaps the template is just putting things in as strings?

[spacecwoboy]

(@andrewvc you bet, amigo -thanks for your work on nmap to ES integration)

I've looked at it pretty well, and from the template - it parses well when a single value is passed - but perhaps the | [pipe] is throwing it off , or accurately defining a new target object, yet it doesnt have a destination object.

If it were possible to dynamically create a variable on input - if os.classes > 1 , and then auto increment/enumerate into the variable names such as:

os.classes.first_guess.{all_fields}  (gen, vendor, accuracy, type, family}
os.classes.second_guess.{all_fields} (gen, vendor, accuracy, type, family}

versus chunking two+ into the same container

os.classes.{first_guess:all_fields,second_guess:all_fields} 

Another option, is to dismiss the second guess (but I dont too much like that idea of dropping data)

Grok'ing and regex can probably yield a workaround if creative.

From the nmap_host portion of the template, the inputs look to be sane - but how to ingest first_guess and second_guess

nmap_host portion of the template:

- os": {
- 					"properties": {
- 						"classes": {
- 							"properties": {
- 								"accuracy": {
- 									"type": "integer"
- 								},
- 								"family": {
- 									"type": "string",
- 									"index": "not_analyzed"
- 								},
- 								"gen": {
- 									"type": "string",
- 									"index": "not_analyzed"
- 								},
- 								"type": {
- 									"type": "string",
- 									"index": "not_analyzed"
- 								},
- 								"vendor": {
- 									"type": "string",
- 									"index": "not_analyzed"
- 								}
- 							}
- 						},
- 						"matches": {
- 							"properties": {
- 								"accuracy": {
- 									"type": "long"
- 								},
- 								"name": {
- 									"type": "string",
- 									"index": "not_analyzed"
- 								}
- 							}

[andrewvc]

I'm a bit confused @spacecwoboy we should be handling the multiple OSes correctly. Are you sure you have the template setup correctly? It should accurately do so here: https://github.com/logstash-plugins/logstash-codec-nmap/blob/master/examples/elasticsearch/elasticsearch_nmap_template.json#L192

I am realizing, however, that we should probably make the OS property nested to allow for boolean queries against multiple OS properties.