blacklanternsecurity
BBOT Freezes
Today, twice my scan froze on tesla and openai. This is the last lines of debug.log
downloaded to /root/.bbot/scans/openai/2024-04-09_17-27-48/filedownload/20240409_1747_02_https-privacy-openai-com-main-penumbra.js
2024-04-09 17:47:07,725 [INFO] bbot.modules.filedownload base.py:1262 Found "footnote-9e6a21e04868e13dc75adf73cfaab1be722d32b46af9a8b1894712c647ab6499.br.js" at "https://global.discourse-cdn.com", downloaded to /root/.bbot/scans/openai/2024-04-09_17-27-48/filedownload/20240409_1747_04_https-global-discourse-cdn-com-openai1-assets-plugins-footnote-9e6a21e04868e13dc75adf73cfaab1be722d32b46af9a8b1894712c647ab6499-br.js
2024-04-09 17:47:09,817 [TRACE] bbot.core.helpers.web web.py:689 Unhandled exception with request to URL: https://github.com/date-fns/date-fns/blob/master/docs/upgradeGuide.md: Client error '404 Not Found' for url 'https://github.com/date-fns/date-fns/blob/main/docs/upgradeGuide.md'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404
BBot v1.1.7.3218rc
Nothing more prints
Another target today, nothing specific in debug.log. Nothing prints more on terminal by bbot
I also got this when stopped it by ctrl + c and it didn't finish properly, so I killed tmux pane.
Also is this a duplicate of https://github.com/blacklanternsecurity/bbot/issues/1053? Or a new issue?
Thanks, this is the debug of openai scan.
This issue is different than #1053 , that one was crashing, but this one just stops and won't print anything else.
I think we've been able to reproduce this. In our case, BBOT was frozen and one CPU core was stuck at 100%. It was completely frozen; we weren't able to cancel it via CTRL+C (which sadly means we weren't able to see what it was doing when it froze).
Based on this I have a couple of guesses as to what this might be:
- A race condition in one of our libraries (in the past we've had similar issues in dnspython and httpx)
- A regex denial-of-service (it's possible there's a certain webpage that's reacting with one of our regexes)
Sadly until we have a reliable way to reproduce this (preferably with a single target + single bbot module), there's not a lot we can do to fix it.
Finally tracked down this bug: https://github.com/blacklanternsecurity/bbot/issues/1414.
We pushed a fix to dev and stable.
BBOT frozen for 15 minutes for now, not sure if it will continue or not. Scan duration was around 4 hours.
Activity detected 20 mins later
Keeps freezing and resuming
Thanks @amiremami. This is most likely an issue with excavate, and will be fixed in bbot-2.0 as soon as https://github.com/blacklanternsecurity/bbot/pull/1471 is merged.
There was a bug with excavate where sometimes its regexes would take a long time.