secdev
maximum recursion depth exceeded with scapy 2.4.4
Im using a Raspberry pi zero w with python 3.7.3, after one day scanning nearby devices i get this error. CODE:
SEEN_DEVICES = {}
d = {'00:00:00:00:00:00':'Example MAC Address'}
IGNORE_LIST = set(['00:00:00:00:00:00', '01:01:01:01:01:01'])
def handle_packet(pkt):
global SEEN_DEVICES
global d
global IGNORE_LIST
if not pkt.haslayer(Dot11ProbeReq) :
return
if pkt.type == 0 and (pkt.subtype == 4 or pkt.subtype == 8): #subtype used to be 8 (APs) but is now 4 (Probe Requests)
curmac = None
curmac = pkt.addr2
curmac = curmac.upper() #Assign variable to packet mac and make it uppercase
if curmac not in IGNORE_LIST: #If not registered as ignored
if curmac in d:
return
else:
signal = pkt.dBm_AntSignal
SEEN_DEVICES[pkt.addr2] = signal
return
return
return
return
#main
parser = argparse.ArgumentParser()
parser.add_argument('--interface', '-i', default='wlan1', help='monitor mode enabled interface')
args = parser.parse_args()
while True:
try:
sniff(iface=args.interface, prn=handle_packet, count = 300,timeout = 2) #start sniffin
print("wifi: ",SEEN_DEVICES)
msg = ""
for dev in SEEN_DEVICES:
msg = msg +"wifi/"+ id + "/" + str(dev) + "/" + str(SEEN_DEVICES[dev]) + " "
SEEN_DEVICES = {}
except Exception as e:
print("wifi")
print(e)
for tb in traceback.format_tb(sys.exc_info()[2]):
print (tb)
sys.exit(1)
ERROR:
Exception ignored in: <function ObjectPipe.__del__ at 0xb59eb228>
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/scapy/automaton.py", line 242, in __del__
self.close()
RecursionError: maximum recursion depth exceeded
Fatal Python error: Cannot recover from stack overflow.
Thread 0xb40ff460 (most recent call first):
File "/usr/local/lib/python3.7/dist-packages/paho/mqtt/client.py", line 1167 in loop
File "/usr/local/lib/python3.7/dist-packages/paho/mqtt/client.py", line 1779 in loop_forever
File "/usr/local/lib/python3.7/dist-packages/paho/mqtt/client.py", line 3452 in _thread_main
File "/usr/lib/python3.7/threading.py", line 865 in run
File "/usr/lib/python3.7/threading.py", line 917 in _bootstrap_inner
File "/usr/lib/python3.7/threading.py", line 885 in _bootstrap
Current thread 0xb4aff460 (most recent call first):
File "/usr/local/lib/python3.7/dist-packages/scapy/layers/dot11.py", line 955 in <lambda>
File "/usr/local/lib/python3.7/dist-packages/scapy/fields.py", line 1380 in getfield
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 839 in do_dissect
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 875 in dissect
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 158 in __init__
File "/usr/local/lib/python3.7/dist-packages/scapy/base_classes.py", line 266 in __call__
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 858 in do_dissect_payload
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 880 in dissect
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 158 in __init__
File "/usr/local/lib/python3.7/dist-packages/scapy/base_classes.py", line 266 in __call__
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 858 in do_dissect_payload
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 880 in dissect
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 158 in __init__
File "/usr/local/lib/python3.7/dist-packages/scapy/base_classes.py", line 266 in __call__
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 858 in do_dissect_payload
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 880 in dissect
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 158 in __init__
File "/usr/local/lib/python3.7/dist-packages/scapy/base_classes.py", line 266 in __call__
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 858 in do_dissect_payload
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 880 in dissect
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 158 in __init__
File "/usr/local/lib/python3.7/dist-packages/scapy/base_classes.py", line 266 in __call__
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 858 in do_dissect_payload
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 880 in dissect
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 158 in __init__
File "/usr/local/lib/python3.7/dist-packages/scapy/base_classes.py", line 266 in __call__
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 858 in do_dissect_payload
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 880 in dissect
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 158 in __init__
File "/usr/local/lib/python3.7/dist-packages/scapy/base_classes.py", line 266 in __call__
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 858 in do_dissect_payload
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 880 in dissect
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 158 in __init__
File "/usr/local/lib/python3.7/dist-packages/scapy/base_classes.py", line 266 in __call__
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 858 in do_dissect_payload
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 880 in dissect
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 158 in __init__
File "/usr/local/lib/python3.7/dist-packages/scapy/base_classes.py", line 266 in __call__
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 858 in do_dissect_payload
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 880 in dissect
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 158 in __init__
File "/usr/local/lib/python3.7/dist-packages/scapy/base_classes.py", line 266 in __call__
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 858 in do_dissect_payload
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 880 in dissect
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 158 in __init__
File "/usr/local/lib/python3.7/dist-packages/scapy/base_classes.py", line 266 in __call__
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 858 in do_dissect_payload
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 880 in dissect
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 158 in __init__
File "/usr/local/lib/python3.7/dist-packages/scapy/base_classes.py", line 266 in __call__
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 858 in do_dissect_payload
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 880 in dissect
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 158 in __init__
File "/usr/local/lib/python3.7/dist-packages/scapy/base_classes.py", line 266 in __call__
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 858 in do_dissect_payload
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 880 in dissect
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 158 in __init__
File "/usr/local/lib/python3.7/dist-packages/scapy/base_classes.py", line 266 in __call__
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 858 in do_dissect_payload
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 880 in dissect
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 158 in __init__
File "/usr/local/lib/python3.7/dist-packages/scapy/base_classes.py", line 266 in __call__
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 858 in do_dissect_payload
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 880 in dissect
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 158 in __init__
File "/usr/local/lib/python3.7/dist-packages/scapy/base_classes.py", line 266 in __call__
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 858 in do_dissect_payload
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 880 in dissect
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 158 in __init__
File "/usr/local/lib/python3.7/dist-packages/scapy/base_classes.py", line 266 in __call__
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 858 in do_dissect_payload
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 880 in dissect
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 158 in __init__
File "/usr/local/lib/python3.7/dist-packages/scapy/base_classes.py", line 266 in __call__
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 858 in do_dissect_payload
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 880 in dissect
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 158 in __init__
File "/usr/local/lib/python3.7/dist-packages/scapy/base_classes.py", line 266 in __call__
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 858 in do_dissect_payload
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 880 in dissect
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 158 in __init__
File "/usr/local/lib/python3.7/dist-packages/scapy/base_classes.py", line 266 in __call__
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 858 in do_dissect_payload
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 880 in dissect
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 158 in __init__
File "/usr/local/lib/python3.7/dist-packages/scapy/base_classes.py", line 266 in __call__
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 858 in do_dissect_payload
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 880 in dissect
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 158 in __init__
File "/usr/local/lib/python3.7/dist-packages/scapy/base_classes.py", line 266 in __call__
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 858 in do_dissect_payload
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 880 in dissect
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 158 in __init__
File "/usr/local/lib/python3.7/dist-packages/scapy/base_classes.py", line 266 in __call__
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 858 in do_dissect_payload
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 880 in dissect
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 158 in __init__
File "/usr/local/lib/python3.7/dist-packages/scapy/base_classes.py", line 266 in __call__
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 858 in do_dissect_payload
File "/usr/local/lib/python3.7/dist-packages/scapy/packet.py", line 880 in dissect
...
Thread 0xb54c2460 (most recent call first):
File "/usr/local/lib/python3.7/dist-packages/bluepy/btle.py", line 342 in _waitResp
File "/usr/local/lib/python3.7/dist-packages/bluepy/btle.py", line 821 in process
File "/usr/local/lib/python3.7/dist-packages/bluepy/btle.py", line 853 in scan
File "scanner.py", line 74 in BleScan
File "/usr/lib/python3.7/threading.py", line 865 in run
File "/usr/lib/python3.7/threading.py", line 917 in _bootstrap_inner
File "/usr/lib/python3.7/threading.py", line 885 in _bootstrap
Thread 0xb6fdb8e0 (most recent call first):
File "/usr/lib/python3.7/threading.py", line 1048 in _wait_for_tstate_lock
File "/usr/lib/python3.7/threading.py", line 1032 in join
File "/usr/lib/python3.7/threading.py", line 1281 in _shutdown
about 38% usage. Im trying to do an infinte loop scanning nearby devices and then with several raspberry pi zero w trilaterate the position in a room.
I will try to isolate the frame but it takes a few days to fail. I think its something in the function handle_packet or in the sniff arguments.
I try to isolate but it doesnt make sense, enter in an if and stops. I think the if isnt the problem but i dont know where is the problem.
if not pkt.haslayer(Dot11ProbeReq) : return
Exception ignored in: <function ObjectPipe.__del__ at 0xb592b2b8> Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/scapy/automaton.py", line 242, in __del__ self.close() RecursionError: maximum recursion depth exceeded Fatal Python error: Cannot recover from stack overflow.
We are experiencing a similar problem in this PR (https://github.com/GoSecure/pyrdp/pull/311) where we reach the stack overflow. We are processing TLS packets over port 3389. We want to leverage scapy to avoid doing the TCP reassembly ourselves.
We will try adding print statements to see if we can figure out when does the infinite recursion (or the really deep one) starts.
When running inside a debugger we get no fatal error but it consumes more and more memory until it is killed by the operating system. 8GB+ of RAM for a 4 MB pcap. Unfortunately, I can't share that pcap. Once we understand what's going on we might be able to create a reproducing pcap that we can share.
We are not able to create a reproducer. This pcap was provided to us and captured in an environment that is different from what we use. There are large frames here and I believe that they might combine many TCP segments in one frame. Wireshark does display everything properly and separately but also shows that the frames are huge.
We switched our approach and tried to use TLSSession instead of TCPSession in the sniff call:
bind_layers(TCP, TLS)
pcap = sniff(offline=str(self.inputFile), session=TLSSession)
We no longer get the stack overflow but we reach a point where packets in the session's sequence are no longer recognized as TLS Application Data.
For example, here is the last packet recognized as TLS Application Data:
ipdb> record.show()
###[ TLS ]###
type = application_data
version = TLS 1.2
len = 11070 [deciphered_len= 2867]
iv = b'\x84_8N\xfduS3'
\msg \
|###[ TLS Application Data ]###
| data = 'ltB\\xab-\\xe8A\\x90\x1e\\xa4\\x87\\x8f\\xc4\\xe1\\xc4\\xd0\\xe9&\\xd1O\\x8eY\\xee\\x82]\x15\x12\\xaa\r\\xf8\x1c\\xa8\\xf
[....]
ipdb> packet[TCP].len
11070
ipdb> len(tcp.payload)
2896
We can see that deciphered length is much smaller than actual TLS length.
The next packet's TCP content of the session is of type _TLSEncryptedContent instead of being reassembled with the previous one or of a TLS application data type:
ipdb> tcp.show()
###[ TCP ]###
sport = ms_wbt_server
dport = 65485
seq = 4166865623
ack = 2207469577
dataofs = 5
reserved = 0
flags = PA
window = 501
chksum = 0x24bf
urgptr = 0
options = []
###[ Encrypted Content ]###
load = '\\xba\\x9a\x16\\xff\\
[...]
ipdb> len(tcp.payload)
2920
Trying to push this further to either provide a fix or find a suitable workaround: should we push forward with making session=TLSSession work in our case or should we investigate more around the stack overflow?
Thanks!
Can you try the master branch?
I just did with current master freshly cloned. The stack recursion crash is gone but there seems to be a recursion or complexity problem still since my Python process was terminated by my OS for consuming too much memory with the following output:
Terminated
Similar to what I previously described here: https://github.com/secdev/scapy/issues/3145#issuecomment-856168900
@obilodeau could you try https://github.com/secdev/scapy/pull/3919? I'm not 100% sure it's the same issue though.
For the record I reproduced the original dot11 stack overflow with Python3.8 on Ubuntu Focal:
Fatal Python error: Cannot recover from stack overflow.
Python runtime state: initialized
Current thread 0x00007fb44cdc2740 (most recent call first):
File "/scapy/scapy/fields.py", line 1389 in any2i
File "/scapy/scapy/packet.py", line 471 in setfieldval
File "/scapy/scapy/layers/dot11.py", line 1019 in __setattr__
File "/scapy/scapy/layers/dot11.py", line 1057 in pre_dissect
File "/scapy/scapy/packet.py", line 1025 in dissect
File "/scapy/scapy/packet.py", line 163 in __init__
File "/scapy/scapy/base_classes.py", line 396 in __call__
File "/scapy/scapy/packet.py", line 1007 in do_dissect_payload
File "/scapy/scapy/packet.py", line 1032 in dissect
File "/scapy/scapy/packet.py", line 163 in __init__
File "/scapy/scapy/base_classes.py", line 396 in __call__
File "/scapy/scapy/packet.py", line 1007 in do_dissect_payload
File "/scapy/scapy/packet.py", line 1032 in dissect
...
It can't be reproduced with Python3.10 and Python3.11 though so I'm inclined to say that it's probably a Python issue where it can't handle the stack overflow for some reason. With Python3.10 and Python3.11 the RecursionError exception is thrown (and caught by the dissector) and can only be seen in action by setting conf.debug_dissector to True. I guess the question is whether it's safe for scapy to rely on Python being able to handle RecursionErrors.
As the oldest version that we currently aim to support is 3.7, I think that we cannot rely on the Python interpreter behavior.
It can't be reproduced with Python3.10 and Python3.11 though so I'm inclined to say that it's probably a Python issue where it can't handle the stack overflow for some reason.
Whether the interpreter handles the stack overflow properly or not is irrelevant imho. A stack overflow is most often caused by infinite recursion, which is certainly an error in the program, or otherwise might indicate that the recursion should better be replaced by iteration or some other approach.
A stack overflow is most often caused by infinite recursion, which is certainly an error in the program
Just to clarify there is no infinite recursion there. It's just that packets with quite a few payloads lead to dissect_payload calling dissect_payload until either all the payloads are parsed or the RecursionError exception is hit. Depending on the number of payloads it can be "fixed" by increasing the recursion limit with sys.setrecursionlimit
otherwise might indicate that the recursion should better be replaced by iteration or some other approach
Agreed. But unfortunately it doesn't seem to be that easy.
Having taken a closer look I think I more or less figured out what's going on here. The issue is that RecursionError doesn't actually bubble up because PacketListField "swallows" it and keeps going instead of recovering from the first RecursionError and it leads to the crash because generally Python doesn't guarantee that it can handle two recursion errors in a row closely following each other. I kind of fixed it with
diff --git a/scapy/fields.py b/scapy/fields.py
index a13e19d..bb99424 100644
--- a/scapy/fields.py
+++ b/scapy/fields.py
@@ -1765,6 +1765,9 @@ class PacketListField(_PacketField[List[BasePacket]]):
p = cls(remain)
else:
p = self.m2i(pkt, remain)
+ except RecursionError:
+ p = conf.raw_layer(load=remain + ret)
+ remain = ret = b""
except Exception:
if conf.debug_dissector:
raise
It works because as soon as the first RecursionError is hit PacketListField consumes all the remaining bytes and all the do_dissect_payload calls complete without triggering new RecursionErrors along the way.
The stack recursion crash is gone but there seems to be a recursion or complexity problem still since my Python process was terminated by my OS for consuming too much memory
With https://github.com/secdev/scapy/pull/3923 merged it should be fixed.
The recursion limit is still hit though but I'm not sure how it can be fixed properly. Apart from iterating over packets somehow I think the safest option would be to catch and propagate RecursionError with raise but it would mean that packets like that wouldn't be dissected at all. Another option would be to catch RecursionError and return, say, None to let the callers know that they should stop parsing anything and just return (it can in theory break though if conf.raw_layer points to some custom layer that can overflow those 50 frames provided by Python on top of the recursion limit to recover). Anyway Python3.10 and Python3.11 don't seem to crash so it should be possible to switch to them to get it around for the time being.
As far as I can remember there were two issues here.
The complexity issue was fixed in https://github.com/secdev/scapy/pull/3923.
The Fatal Python error: Cannot recover from stack overflow issue can be hit with Python3.8 but I haven't seen it with 3.9, 3.10, 3.11 or 3.12 in practice. Ideally RecursionError should be handled somehow (https://github.com/secdev/scapy/issues/3145#issuecomment-1456320141) to avoid relying on "undefined" behavior of the Python implementation but unfortunately I haven't figured out how to do that. Then again in practice currently it doesn't crash with the latest versions of cpython at least.
Okay. We'll close this and investigate more closely if it ever comes up again. Thanks a lot