About postprocessing JSON output

[joydragon]

This is really a request for an enhancement

Can you post-process the JSON (and XML) output so the IPs get all joined together on the scan?

For example, the following code: $masscan -p21,22,80,443 10.0.0.1/24 -oJ test.json

Gets me something like:

$cat test.json
[
{"ip":"10.0.0.100", "timestamp":"1516653779", "ports": [{"port": 22, "proto": "tcp", "status": "open", "reason": "syn-ack", "ttl": 128}]},
{"ip":"10.0.0.100", "timestamp":"1516653780", "ports": [{"port": 80, "proto": "tcp", "status": "open", "reason": "syn-ack", "ttl": 128}]},
{"ip":"10.0.0.101", "timestamp":"1516653780", "ports": [{"port": 80, "proto": "tcp", "status": "open", "reason": "syn-ack", "ttl": 128}]},
{"ip":"10.0.0.101", "timestamp":"1516653781", "ports": [{"port": 443, "proto": "tcp", "status": "open", "reason": "syn-ack", "ttl": 128}]}
]

To get what I want I have to use jq, like this:

$cat test.json | jq 'group_by(.ip) | map({"ip": .[0].ip, "timestamp": map(.timestamp), "ports": map(.ports[0])})'
[
{"ip":"10.0.0.100", "timestamp":["1516653779","1516653780"], "ports": [{"port": 22, "proto": "tcp", "status": "open", "reason": "syn-ack", "ttl": 128},{"port": 80, "proto": "tcp", "status": "open", "reason": "syn-ack", "ttl": 128}]},
{"ip":"10.0.0.101", "timestamp":["1516653780","1516653781"], "ports": [{"port": 80, "proto": "tcp", "status": "open", "reason": "syn-ack", "ttl": 128},{"port": 443, "proto": "tcp", "status": "open", "reason": "syn-ack", "ttl": 128}]}
]

I know that it might be impossible to do it on the scan, because of the asynchronous nature, but can it be post-processed somehow? Or should I keep using the jq command?

Thanks!

[anarchy1337x]

You are a livesaver for that command. it's gonna make it really easy to add to a database, thanks!

[p-l-]

Sorry to interrupt with a bit of self-promotion, but if you are using Masscan for large (and possibly, sharded) scans, you may need a database for that. IVRE can be used here (it relies on MongoDB but can also use the TinyDB module for a limited number of hosts).

It can merge scan results (from Masscan, Nmap, ZGrab2 and several other tools) into a "view" that you can then browse (using a Web UI or CLI tools) and export as JSON or XML (Nmap XML format).

See IVRE Web site and the repository. Feel free to come ask your questions on Gitter.

Hope this helps!