lzr
lzr copied to clipboard
stanford-esrg
it's acting weird!
Hello there, I'm running lzr with zmap, it gives me correct output for one time and then it returns 0 results for all other scans!
note: my main interface is ens160, there is only one IP address on this interface, I've used this to make the lzr make all source-ip=x.x.x.x/32
my target network has at least 21 results on port 80, zmap can verify that but I don't know what happens that it's giving me this results !
one time that it gave me results ( just 5 item, it should give me about 21 items ) !
{"saddr":"x.x.x.13","daddr":"185.x.x.x","sport":80,"dport":56425,"seqnum":3014031342,"acknum":110,"window":29200,"ttl":60,"Counter":0,"ACK":true,"ACKed":true,"SYN":false,"RST":false,"FIN":false,"PUSH":true,"HandshakeNum":1,"fingerprint":"http","Timestamp":"2023-11-13T12:37:18.393362549+03:30","expectedRToLZR":"data","data":"HTTP/1.1 301 Moved Permanently\r\nServer: openresty\r\nDate: Mon, 13 Nov 2023 09:07:18 GMT\r\nContent-Type: text/html\r\nContent-Length: 166\r\nConnection: keep-alive\r\nLocation: https://x.x.x.13/\r\n\r\n\u003chtml\u003e\r\n\u003chead\u003e\u003ctitle\u003e301 Moved Permanently\u003c/title\u003e\u003c/head\u003e\r\n\u003cbody\u003e\r\n\u003ccenter\u003e\u003ch1\u003e301 Moved Permanently\u003c/h1\u003e\u003c/center\u003e\r\n\u003chr\u003e\u003ccenter\u003eopenresty\u003c/center\u003e\r\n\u003c/body\u003e\r\n\u003c/html\u003e\r\n"} {"saddr":"x.x.x.12","daddr":"185.x.x.x","sport":80,"dport":46649,"seqnum":1393460254,"acknum":110,"window":29200,"ttl":60,"Counter":0,"ACK":true,"ACKed":true,"SYN":false,"RST":false,"FIN":false,"PUSH":true,"HandshakeNum":1,"fingerprint":"http","Timestamp":"2023-11-13T12:37:18.393380333+03:30","expectedRToLZR":"data","data":"HTTP/1.1 301 Moved Permanently\r\nServer: openresty\r\nDate: Mon, 13 Nov 2023 09:07:18 GMT\r\nContent-Type: text/html\r\nContent-Length: 166\r\nConnection: keep-alive\r\nLocation: https://x.x.x.12/\r\n\r\n\u003chtml\u003e\r\n\u003chead\u003e\u003ctitle\u003e301 Moved Permanently\u003c/title\u003e\u003c/head\u003e\r\n\u003cbody\u003e\r\n\u003ccenter\u003e\u003ch1\u003e301 Moved Permanently\u003c/h1\u003e\u003c/center\u003e\r\n\u003chr\u003e\u003ccenter\u003eopenresty\u003c/center\u003e\r\n\u003c/body\u003e\r\n\u003c/html\u003e\r\n"} {"saddr":"x.x.x.11","daddr":"185.x.x.x","sport":80,"dport":47539,"seqnum":2199870258,"acknum":110,"window":29200,"ttl":60,"Counter":0,"ACK":true,"ACKed":true,"SYN":false,"RST":false,"FIN":false,"PUSH":true,"HandshakeNum":1,"fingerprint":"http","Timestamp":"2023-11-13T12:37:18.393385312+03:30","expectedRToLZR":"data","data":"HTTP/1.1 301 Moved Permanently\r\nServer: openresty\r\nDate: Mon, 13 Nov 2023 09:07:18 GMT\r\nContent-Type: text/html\r\nContent-Length: 166\r\nConnection: keep-alive\r\nLocation: https://x.x.x.11/\r\n\r\n\u003chtml\u003e\r\n\u003chead\u003e\u003ctitle\u003e301 Moved Permanently\u003c/title\u003e\u003c/head\u003e\r\n\u003cbody\u003e\r\n\u003ccenter\u003e\u003ch1\u003e301 Moved Permanently\u003c/h1\u003e\u003c/center\u003e\r\n\u003chr\u003e\u003ccenter\u003eopenresty\u003c/center\u003e\r\n\u003c/body\u003e\r\n\u003c/html\u003e\r\n"} {"saddr":"x.x.x.10","daddr":"185.x.x.x","sport":80,"dport":51667,"seqnum":2846776825,"acknum":110,"window":29200,"ttl":60,"Counter":0,"ACK":true,"ACKed":true,"SYN":false,"RST":false,"FIN":false,"PUSH":true,"HandshakeNum":1,"fingerprint":"http","Timestamp":"2023-11-13T12:37:18.393396227+03:30","expectedRToLZR":"data","data":"HTTP/1.1 301 Moved Permanently\r\nServer: openresty\r\nDate: Mon, 13 Nov 2023 09:07:18 GMT\r\nContent-Type: text/html\r\nContent-Length: 166\r\nConnection: keep-alive\r\nLocation: https://x.x.x.10/\r\n\r\n\u003chtml\u003e\r\n\u003chead\u003e\u003ctitle\u003e301 Moved Permanently\u003c/title\u003e\u003c/head\u003e\r\n\u003cbody\u003e\r\n\u003ccenter\u003e\u003ch1\u003e301 Moved Permanently\u003c/h1\u003e\u003c/center\u003e\r\n\u003chr\u003e\u003ccenter\u003eopenresty\u003c/center\u003e\r\n\u003c/body\u003e\r\n\u003c/html\u003e\r\n"} {"saddr":"x.x.x.14","daddr":"185.x.x.x","sport":80,"dport":57849,"seqnum":1098684816,"acknum":110,"window":29200,"ttl":60,"Counter":0,"ACK":true,"ACKed":true,"SYN":false,"RST":false,"FIN":false,"PUSH":true,"HandshakeNum":1,"fingerprint":"http","Timestamp":"2023-11-13T12:37:18.39340616+03:30","expectedRToLZR":"data","data":"HTTP/1.1 301 Moved Permanently\r\nServer: openresty\r\nDate: Mon, 13 Nov 2023 09:07:18 GMT\r\nContent-Type: text/html\r\nContent-Length: 166\r\nConnection: keep-alive\r\nLocation: https://x.x.x.14/\r\n\r\n\u003chtml\u003e\r\n\u003chead\u003e\u003ctitle\u003e301 Moved Permanently\u003c/title\u003e\u003c/head\u003e\r\n\u003cbody\u003e\r\n\u003ccenter\u003e\u003ch1\u003e301 Moved Permanently\u003c/h1\u003e\u003c/center\u003e\r\n\u003chr\u003e\u003ccenter\u003eopenresty\u003c/center\u003e\r\n\u003c/body\u003e\r\n\u003c/html\u003e\r\n"}
output for wrong results: root@ubuntu20:/tmp/lzr# port=80 ; target=86.x.x.x/24 ; zmap -c 5 --retries=3 $target -i ens160 --target-port=$port --output-filter="success = 1 && repeat = 0" -f "saddr,daddr,sport,dport,seqnum,acknum,window" -O json | lzr --handshakes wait,http,tls -sendSYNs -sendInterface ens160 Nov 13 13:05:50.540 [WARN] blocklist: ZMap is currently using the default blocklist located at /etc/zmap/blocklist.conf. By default, this blocklist excludes locally scoped networks (e.g. 10.0.0.0/8, 127.0.0.1/8, and 192.168.0.0/16). If you are trying to scan local networks, you can change the default blocklist by editing the default ZMap configuration at /etc/zmap/blocklist.conf. If you have modified the default blocklist, you can ignore this message. Nov 13 13:05:50.542 [INFO] dedup: Response deduplication method is full ++Writing results to file: default_20231113130550.json ++Handshakes: wait,http,tls ++Sending SYNs ++Using Sending Interface: ens160 ++Worker threads: 1 ++Timeout Interval (s): 5 ++Retransmit Interval (s): 1 ++Number of Retransmitions: 1 Nov 13 13:05:50.617 [INFO] recv: duplicate responses will be passed to the output module Nov 13 13:05:50.620 [INFO] recv: unsuccessful responses will be passed to the output module 0:00 0%; send: 0 0 p/s (0 p/s avg); recv: 0 0 p/s (0 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.00% 0:00 0%; send: 0 0 p/s (0 p/s avg); recv: 0 0 p/s (0 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.00% 0:01 21%; send: 256 done (2.49 Kp/s avg); recv: 21 21 p/s (19 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 8.20% 0:02 40%; send: 256 done (2.49 Kp/s avg); recv: 21 0 p/s (10 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 8.20% 0:03 60%; send: 256 done (2.49 Kp/s avg); recv: 21 0 p/s (6 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 8.20% 0:04 80%; send: 256 done (2.49 Kp/s avg); recv: 21 0 p/s (5 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 8.20% 0:05 99% (1s left); send: 256 done (2.49 Kp/s avg); recv: 21 0 p/s (4 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 8.20% Nov 13 13:05:56.633 [INFO] zmap: completed Killed
I even created services.txt in this format and I've verified all of them has port 80 open: IP:port
cat services.txt | pv -L3 -l --qui | lzr --handshakes wait,http,tls -sendSYNs -sendInterface ens160 -gatewayMac xx:xx:71:f7 -sourceIP 185.x.x.x
and here is the results:
++Writing results to file: default_20231113132411.json ++Handshakes: wait,http,tls ++Sending SYNs ++Using SourceIP: true ++Using Sending Interface: ens160 ++Using Gateway Mac: xxxxx ++Worker threads: 1 ++Timeout Interval (s): 5 ++Retransmit Interval (s): 1 ++Number of Retransmitions: 1 Finished Reading Input Runtime: 7.448603923s {"TotalResponses":0,"ZeroWindow":0,"ACKed":0,"Data":0,"No_SYNACK":0,"Rst":0,"Fin":0,"Resp_ack":0,"HyperACKtive":0}
Hi! Can you take a look at the issue I linked below and read through it and 1. take similar debugging steps as I recommended there and 2. see if the eventual solution there is applicable to your case?
https://github.com/stanford-esrg/lzr/issues/13
look we have 52% hitrate based on Zmap output, I can't figureout what's happening that LZR reacts like this
root@ubuntu20:~# sudo zmap 185.x.x.0/24 --target-port=80 --output-filter="success = 1 && repeat = 0" -f "saddr,daddr,sport,dport,seqnum,acknum,window" -O json --source-ip=185.x.x.x -i ens160 | sudo lzr --handshakes wait,http,tls -sendInterface ens160 -gatewayMac xxx:b3:71:f7 -feedZGrab | zgrab2 multiple -c /tmp/zgrab2/testrun/base-configurations/all.ini Nov 14 09:23:49.289 [WARN] blocklist: ZMap is currently using the default blocklist located at /etc/zmap/blocklist.conf. By default, this blocklist excludes locally scoped networks (e.g. 10.0.0.0/8, 127.0.0.1/8, and 192.168.0.0/16). If you are trying to scan local networks, you can change the default blocklist by editing the default ZMap configuration at /etc/zmap/blocklist.conf. If you have modified the default blocklist, you can ignore this message. Nov 14 09:23:49.291 [INFO] dedup: Response deduplication method is full INFO[0000] started grab at 2023-11-14T09:23:49+03:30 ++Writing results to file: default_20231114092349.json ++Handshakes: wait,http,tls ++Using Sending Interface: ens160 ++Using Gateway Mac: xxxx3:71:f7 ++Feeding ZGrab with fingerprints ++Worker threads: 1 ++Timeout Interval (s): 5 ++Retransmit Interval (s): 1 ++Number of Retransmitions: 1 Nov 14 09:23:49.391 [INFO] recv: duplicate responses will be passed to the output module Nov 14 09:23:49.391 [INFO] recv: unsuccessful responses will be passed to the output module 0:00 0%; send: 0 0 p/s (0 p/s avg); recv: 0 0 p/s (0 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.00% 0:01 13%; send: 256 done (2.25 Kp/s avg); recv: 0 0 p/s (0 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.00% 0:02 25%; send: 256 done (2.25 Kp/s avg); recv: 134 134 p/s (64 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 52.34% 0:03 38%; send: 256 done (2.25 Kp/s avg); recv: 134 0 p/s (43 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 52.34% 0:04 50%; send: 256 done (2.25 Kp/s avg); recv: 134 0 p/s (32 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 52.34% 0:05 62% (4s left); send: 256 done (2.25 Kp/s avg); recv: 134 0 p/s (26 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 52.34% 0:06 75% (3s left); send: 256 done (2.25 Kp/s avg); recv: 134 0 p/s (22 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 52.34% 0:07 87% (2s left); send: 256 done (2.25 Kp/s avg); recv: 134 0 p/s (18 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 52.34% 0:08 99% (1s left); send: 256 done (2.25 Kp/s avg); recv: 134 0 p/s (16 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 52.34% Nov 14 09:23:58.400 [INFO] zmap: completed Finished Reading Input Runtime: 9.130180185s {"TotalResponses":0,"ZeroWindow":0,"ACKed":0,"Data":0,"No_SYNACK":0,"Rst":0,"Fin":0,"Resp_ack":0,"HyperACKtive":0} INFO[0009] finished grab at 2023-11-14T09:23:58+03:30 {"statuses":{},"start":"2023-11-14T09:23:49+03:30","end":"2023-11-14T09:23:58+03:30","duration":"9.399893063s"}
I've used the release version instead of master and now it's kinda working
root@xxx:/home/xxx/zmap_R_D/lzr/v1/lzr-1/cmd/lzr# sudo zmap x.x.x.20/30 --target-port=80 --output-filter="success = 1 && repeat = 0" -f "saddr,daddr,sport,dport,seqnum,acknum,window" -O json --source-ip=x.x.x.3 -i ens160 | sudo ./lzr --handshakes wait,http,tls -sendInterface ens160 -gatewayMac 0xxxx6:07:e4:61 -feedZGrab | zgrab2 multiple -c /home/xxx/zmap_R_D/zgrab2/zgrab2-configurations/base-configurations/all.ini ++Writing results to file: default_20231115100922.json ++Handshakes: wait,http,tls ++Using Sending Interface: ens160 ++Using Gateway Mac: 00:50:56:07:e4:61 ++Feeding ZGrab with fingerprints ++Worker threads: 1 ++Timeout Interval (s): 5 ++Retransmit Interval (s): 1 ++Number of Retransmitions: 1 INFO[0000] started grab at 2023-11-15T10:09:22Z Nov 15 10:09:22.984 [INFO] recv: duplicate responses will be passed to the output module Nov 15 10:09:22.984 [INFO] recv: unsuccessful responses will be passed to the output module 0:00 1%; send: 1 1 p/s (22 p/s avg); recv: 0 0 p/s (0 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.00% 0:01 13%; send: 4 done (91 p/s avg); recv: 4 4 p/s (3 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 100.00% 0:02 25%; send: 4 done (91 p/s avg); recv: 4 0 p/s (1 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 100.00% 0:03 38%; send: 4 done (91 p/s avg); recv: 4 0 p/s (1 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 100.00% 0:04 50%; send: 4 done (91 p/s avg); recv: 4 0 p/s (0 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 100.00% 0:05 63% (3s left); send: 4 done (91 p/s avg); recv: 4 0 p/s (0 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 100.00% 0:06 75% (2s left); send: 4 done (91 p/s avg); recv: 4 0 p/s (0 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 100.00% 0:07 88% (1s left); send: 4 done (91 p/s avg); recv: 4 0 p/s (0 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 100.00% 0:08 101% (0s left); send: 4 done (91 p/s avg); recv: 4 0 p/s (0 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 100.00% Nov 15 10:09:32.032 [INFO] zmap: completed Finished Reading Input Finishing Last: 4 Finishing Last: 4 Finishing Last: 4 Finishing Last: 4 Finishing Last: 4 Finishing Last: 4 Finishing Last: 4 Finishing Last: 4 Finishing Last: 4 Finishing Last: 4 Finishing Last: 4 Finishing Last: 4 Finishing Last: 4 Finishing Last: 4 Finishing Last: 4 Finishing Last: 4 Finishing Last: 4 Finishing Last: 0 Runtime: 27.127366037s {"TotalResponses":4,"ZeroWindow":0,"ACKed":0,"Data":0,"No_SYNACK":4,"Rst":0,"Fin":0,"Resp_ack":0,"HyperACKtive":0} unknown: 4 {"ip":"x.x.x.20"} {"ip":"x.x.x.22"} {"ip":"x.x.x.21"} {"ip":"x.x.x.23"} INFO[0027] finished grab at 2023-11-15T10:09:50Z {"statuses":{},"start":"2023-11-15T10:09:22Z","end":"2023-11-15T10:09:50Z","duration":"27.236364744s"}
but it couldn't detect HTTP, it says unknown also I don't know what are these :
Finished Reading Input Finishing Last: 4 Finishing Last: 4 Finishing Last: 4 Finishing Last: 4 Finishing Last: 4 Finishing Last: 4
didn't saw them before on LZR
here is the output file content
root@xxx:/home/xxx/zmap_R_D/lzr/v1/lzr-1/cmd/lzr# cat default_20231115100922.json {"saddr":"x.x.x.20","daddr":"x.x.x.3","sport":80,"dport":41528,"seqnum":0,"acknum":0,"window":29200,"ttl":0,"Counter":1,"ACK":false,"ACKed":false,"SYN":false,"RST":false,"FIN":false,"PUSH":false,"HandshakeNum":2,"fingerprint":"unknown","Timestamp":"2023-11-15T10:09:44.961352982Z","expectedRToLZR":"sa"} {"saddr":"x.x.x.22","daddr":"x.x.x.3","sport":80,"dport":36285,"seqnum":0,"acknum":0,"window":29200,"ttl":0,"Counter":1,"ACK":false,"ACKed":false,"SYN":false,"RST":false,"FIN":false,"PUSH":false,"HandshakeNum":2,"fingerprint":"unknown","Timestamp":"2023-11-15T10:09:44.961360806Z","expectedRToLZR":"sa"} {"saddr":"x.x.x.21","daddr":"x.x.x.3","sport":80,"dport":49518,"seqnum":0,"acknum":0,"window":29200,"ttl":0,"Counter":1,"ACK":false,"ACKed":false,"SYN":false,"RST":false,"FIN":false,"PUSH":false,"HandshakeNum":2,"fingerprint":"unknown","Timestamp":"2023-11-15T10:09:44.961365646Z","expectedRToLZR":"sa"} {"saddr":"x.x.x.23","daddr":"x.x.x.3","sport":80,"dport":59933,"seqnum":0,"acknum":0,"window":29200,"ttl":0,"Counter":1,"ACK":false,"ACKed":false,"SYN":false,"RST":false,"FIN":false,"PUSH":false,"HandshakeNum":2,"fingerprint":"unknown","Timestamp":"2023-11-15T10:09:44.961375023Z","expectedRToLZR":"sa"}
Here I've used -sendsyn + source ip too, but it gave me error
root@xxxx:/home/xxxx/zmap_R_D/lzr/v1/lzr-1/cmd/lzr# sudo zmap x.x.x.1/30 --target-port=80 --output-filter="success = 1 && repeat = 0" -f "saddr,daddr,sport,dport,seqnum,acknum,window" -O json --source-ip=x.x.x.3 -i ens160 | sudo ./lzr -sendSYNs -sourceIP x.x.x.3 --handshakes wait,http,tls -sendInterface ens160 -gatewayMac xxxx:07:e4:61 -feedZGrab | zgrab2 multiple -c /home/xxxx/zmap_R_D/zgrab2/zgrab2-configurations/base-configurations/all.ini ++Writing results to file: default_20231115101434.json ++Handshakes: wait,http,tls ++Sending SYNs ++Using SourceIP: true ++Using Sending Interface: ens160 ++Using Gateway Mac: xxxx:07:e4:61 ++Feeding ZGrab with fingerprints ++Worker threads: 1 ++Timeout Interval (s): 5 ++Retransmit Interval (s): 1 ++Number of Retransmitions: 1 INFO[0000] started grab at 2023-11-15T10:14:34Z Nov 15 10:14:34.300 [INFO] recv: duplicate responses will be passed to the output module Nov 15 10:14:34.301 [INFO] recv: unsuccessful responses will be passed to the output module 0:00 0%; send: 1 1 p/s (36 p/s avg); recv: 0 0 p/s (0 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.00% panic: Error parsing input list
goroutine 54 [running]: github.com/stanford-esrg/lzr.convertFromInputListToPacket({0xc5374dacf0, 0x81}) /home/xxxx/zmap_R_D/lzr/v1/lzr-1/packet.go:152 +0x3e5 github.com/stanford-esrg/lzr.ConstructIncomingRoutine.func1() /home/xxxx/zmap_R_D/lzr/v1/lzr-1/construct_main_routines.go:73 +0x185 created by github.com/stanford-esrg/lzr.ConstructIncomingRoutine in goroutine 1 /home/xxxx/zmap_R_D/lzr/v1/lzr-1/construct_main_routines.go:58 +0x66 0:01 13%; send: 4 done (145 p/s avg); recv: 4 4 p/s (3 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 100.00% INFO[0001] finished grab at 2023-11-15T10:14:35Z {"statuses":{},"start":"2023-11-15T10:14:34Z","end":"2023-11-15T10:14:35Z","duration":"1.10129257s"} 0:02 25%; send: 4 done (145 p/s avg); recv: 4 0 p/s (1 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 100.00% 0:03 38%; send: 4 done (145 p/s avg); recv: 4 0 p/s (1 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 100.00% 0:04 50%; send: 4 done (145 p/s avg); recv: 4 0 p/s (0 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 100.00% 0:05 63% (3s left); send: 4 done (145 p/s avg); recv: 4 0 p/s (0 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 100.00% 0:06 75% (2s left); send: 4 done (145 p/s avg); recv: 4 0 p/s (0 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 100.00% 0:07 88% (1s left); send: 4 done (145 p/s avg); recv: 4 0 p/s (0 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 100.00% 0:08 100% (0s left); send: 4 done (145 p/s avg); recv: 4 0 p/s (0 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 100.00% Nov 15 10:14:43.302 [INFO] zmap: completed