spiderfoot
spiderfoot copied to clipboard
smicallef
Spiderfoot freezes after about an hour, no error
Please provide a description of the issue and any relevant error messages. The scan just freezes. I am trying to run a scan on a fresh Ubuntu installation and after about an hour it just freezes and I get nothing for several hours. No error message given as it just freezes. I see that python3 becomes a zombie process though.
If you can provide an application stack trace that is even better. How do I do this?
What version of Python are you using? Python 3.8.10
What version of SpiderFoot are you using (stable release or Git master branch)?
Git master branch
You may also wish to check if your issue has been posted previously: I didn't see anything, I hope I did not miss anything.
- https://github.com/smicallef/spiderfoot/issues
Are you able to check and share the logs in $HOME/.spiderfoot/logs/spiderfoot.debug.log to see what was happening before the freeze?
What does "freezes" mean in this context? Is the web interface still responsive? DId the system show high CPU usage?
What kind of scan were you running? A use case scan (Passive, Footprint, Investigate, All) ?
My guess is that the sfp_binstring module is the cause. This module can take a while to run and shows no output while doing so. But it is impossible to know without logs.
The contents of $HOME/.spiderfoot/logs/spiderfoot.debug.log would be useful to see what the scan was doing - in particular the last 50 lines or so. Note that the log is rotated. As it has been a few days, the spiderfoot.debug.log file has likely rotated and the file name now ends with a datestamp.
@smicallef I will see about getting logs this week. I have tried to run in on my work VM which the scan froze after about 30 hours. I ran it on both a ubuntu and Debian system also and they froze after about an hour or two.
@bcoles Freeze as in the output wasn't moving at all for over 5 hours. The web GUI still worked but the modules running just were not running any more. CPU usage was high since I started a couple scans on the work VM, but I was doing a single scan on both Digital Ocean boxes. Setup with the $15/month plan. It has frozen with both the All and selected modules that I have api keys for. I also set the settings to do 20 modules at one time, then thought that was too much so switched it to 15.
It has frozen with both the All and selected modules that I have api keys for. I also set the settings to do 20 modules at one time, then thought that was too much so switched it to 15.
My guess, for now, is still the stp_binstring module, but if that wasn't included in "selected modules that I have api keys for" then I'm not sure.
I suggest using less than 15 modules at once, as a handful of modules manage their own threads, which means that in a worst case scenario you can end up with a few hundred threads running concurrently.
@bcoles, I feel pretty confident to say that I was probably using the stp_binstring even with the API module scan I did as I added pretty much all the tools and only added one bad DNS. The scan I am running right now from my personal computer as a test has been running with 15 modules for about 48 hours right now - set to scan All.
If you suggest less than 15 modules, what would be a good amount for you if only running one scan at a time? That could be the issue with the laptop as I was trying to run a couple scans with 15-20 modules selected at a time and it was pegging the CPU at about 90-100% for an entire day.
How long should I expect on average for an All scan to take?
That all sounds like behaviour exhibited by sfp_binstring. I suggest trying without it. I also strongly suggest taking a look at the logs.
How long should I expect on average for an All scan to take?
Typically scans take less than a day to 5 days depending on the size of the target. Most of my scans finish within 2 days.
However I rarely use All modules, preferring to use Passive modules instead. All is not necessary for my workflow.
I've tested the non-commercial Passive modules extensively. The Footprint and Investigate modules not so much.
If you're comfortably messing around with changing settings and are keen to run some tests, I suggest trying scans without sfp_binstring. For quick sanity check tests, I usually use sfp_dnsresolve and a handful of other modules.
I haven't worked on developing SpiderFoot for a while and I'm still catching up with the changes introduced in version 4.0. The sfp_names module is causing unusual behaviour for me and I'm not sure why. You could try running without that module too.
If you suggest less than 15 modules, what would be a good amount for you if only running one scan at a time?
Unfortunately, due to the aforementioned issue with threads, a "safe" "good" number of max threads is the default of 3.
Alternatively, you can manually configure the settings for each of the modules which manage their own threads:
# grep -rn maxthreads modules/*.py
modules/sfp_accounts.py:41: "_maxthreads": 20
modules/sfp_accounts.py:52: "_maxthreads": "Maximum threads"
modules/sfp_accounts.py:185: for i in range(min(len(sites), self.opts['_maxthreads'])):
modules/sfp_azureblobstorage.py:38: "_maxthreads": 20
modules/sfp_azureblobstorage.py:44: "_maxthreads": "Maximum threads"
modules/sfp_azureblobstorage.py:114: if i >= self.opts['_maxthreads']:
modules/sfp_digitaloceanspace.py:39: "_maxthreads": 20
modules/sfp_digitaloceanspace.py:46: "_maxthreads": "Maximum threads"
modules/sfp_digitaloceanspace.py:131: if i >= self.opts['_maxthreads']:
modules/sfp_dnsbrute.py:39: "_maxthreads": 100
modules/sfp_dnsbrute.py:50: "_maxthreads": "Maximum threads"
modules/sfp_dnsbrute.py:207: if len(targetList) <= self.opts['_maxthreads']:
modules/sfp_dnsbrute.py:232: if len(list(nextsubs.keys())) >= self.opts['_maxthreads']:
modules/sfp_googleobjectstorage.py:38: "_maxthreads": 20
modules/sfp_googleobjectstorage.py:44: "_maxthreads": "Maximum threads"
modules/sfp_googleobjectstorage.py:129: if i >= self.opts['_maxthreads']:
modules/sfp_portscan_tcp.py:43: 'maxthreads': 10,
modules/sfp_portscan_tcp.py:51: 'maxthreads': "Number of ports to try to open simultaneously (number of threads to spawn at once.)",
modules/sfp_portscan_tcp.py:225: if i < self.opts['maxthreads']:
modules/sfp_s3bucket.py:39: "_maxthreads": 20
modules/sfp_s3bucket.py:46: "_maxthreads": "Maximum threads"
modules/sfp_s3bucket.py:129: if i >= self.opts['_maxthreads']:
modules/sfp_tldsearch.py:37: '_maxthreads': 50
modules/sfp_tldsearch.py:44: "_maxthreads": "Maximum threads"
modules/sfp_tldsearch.py:192: if len(targetList) <= self.opts['_maxthreads']:
A worst case scenario would be all of those modules running at once, totaling 260 threads. Additionally, as these are all long-running modules, that significantly increases the likelihood of a worst case scenario as all other faster modules will complete, resulting in these all running concurrently. Fortunately most of these are DNS related modules which requires minimal bandwidth and system resources.
Lowering these to 10 threads each would allow setting a higher number of max threads for SpiderFoot.
@bcoles, thanks for the detailed information. I will bump down my concurrent modules then and see if that helps. I wasn't thinking a single module could run for so long with no output. The client I am running these for wants a pretty deep OSINT inspection. The all is probably still too much but I would rather have to much than not enough. I will respond back on Monday after I have time to look through the logs.
I will bump down my concurrent modules then and see if that helps.
I strongly suggest reviewing the logs to determine the root cause, else you may be in the exact same situation a few days from now.
I also suggest enabling debugging in the Settings tab if you haven't already.
The client I am running these for wants a pretty deep OSINT inspection. The all is probably still too much but I would rather have to much than not enough. I will respond back on Monday after I have time to look through the logs.
SpiderFoot is a good first step for an investigation, but is not even close to a deep investigation. I imagine your manual investigation workflows will cover much of the functionality you would miss by running only a Passive scan. For instance, anything that binstring would have found you would find during manual review of document metadata, decompiling APKs, running strings on executables, etc.
For reference, here's the modules you will miss by running only a Passive scan.
# grep -rn useCase modules/*.py | fgrep -v Passive
modules/sfp_ahmia.py:28: 'useCases': ["Footprint", "Investigate"],
modules/sfp_binstring.py:24: 'useCases': ["Footprint"],
modules/sfp_crossref.py:27: 'useCases': ["Footprint"],
modules/sfp_darksearch.py:29: 'useCases': ["Footprint", "Investigate"],
modules/sfp_dnsbrute.py:27: 'useCases': ["Footprint", "Investigate"],
modules/sfp_dnscommonsrv.py:25: 'useCases': ["Footprint", "Investigate"],
modules/sfp_dnsneighbor.py:25: 'useCases': ["Footprint", "Investigate"],
modules/sfp_dnszonexfer.py:27: 'useCases': ["Footprint", "Investigate"],
modules/sfp_filemeta.py:34: 'useCases': ["Footprint"],
modules/sfp_hackertarget.py:32: 'useCases': ["Footprint", "Investigate"],
modules/sfp_junkfiles.py:24: 'useCases': ["Footprint"],
modules/sfp_onioncity.py:25: 'useCases': ["Footprint", "Investigate"],
modules/sfp_onionsearchengine.py:28: 'useCases': ["Footprint", "Investigate"],
modules/sfp_portscan_tcp.py:29: 'useCases': ["Footprint", "Investigate"],
modules/sfp_similar.py:67: 'useCases': ["Footprint", "Investigate"],
modules/sfp_spider.py:26: 'useCases': ["Footprint", "Investigate"],
modules/sfp_sslcert.py:24: 'useCases': ["Footprint", "Investigate"],
modules/sfp_subdomain_takeover.py:26: 'useCases': ["Footprint", "Investigate"],
modules/sfp_tldsearch.py:29: 'useCases': ["Footprint"],
modules/sfp_tool_cmseek.py:28: 'useCases': ["Footprint", "Investigate"],
modules/sfp_tool_dnstwist.py:28: 'useCases': ["Footprint", "Investigate"],
modules/sfp_tool_nbtscan.py:28: "useCases": ["Footprint", "Investigate"],
modules/sfp_tool_nmap.py:27: 'useCases': ["Footprint", "Investigate"],
modules/sfp_tool_nuclei.py:34: "useCases": [
modules/sfp_tool_onesixtyone.py:29: "useCases": ["Footprint", "Investigate"],
modules/sfp_tool_retirejs.py:29: "useCases": [
modules/sfp_tool_snallygaster.py:28: "useCases": ["Footprint", "Investigate"],
modules/sfp_tool_testsslsh.py:30: 'useCases': ["Footprint", "Investigate"],
modules/sfp_tool_trufflehog.py:28: 'useCases': ["Footprint", "Investigate"],
modules/sfp_tool_wafw00f.py:26: 'useCases': ["Footprint", "Investigate"],
modules/sfp_tool_wappalyzer.py:28: "useCases": [
modules/sfp_tool_webtech.py:26: 'useCases': ["Footprint", "Investigate"],
modules/sfp_tool_whatweb.py:27: 'useCases': ["Footprint", "Investigate"],
modules/sfp_torch.py:26: 'useCases': ["Footprint", "Investigate"],
Also, half of these are sfp_tool modules which won't run unless you have configured the tools.
Further investigation reveals that there is in fact a bug related to the sfp_names module (caused by the sfp_stackoverflow module) which may or may not cause a hang and high CPU usage.
It's one of those fun bugs that occurs on some systems but not others. Maybe it is racey. On one of my systems (with less cores) it causes 100% CPU usage. This system is spec'd very similarly to the $15 digital ocean box (2 cores; 2GB RAM).
On two other system this issue does not occur (4 cores/6GB RAM; 6 cores/8GB RAM).
I'll post in this issue when it is patched. As a temporary workaround you could try running a scan without the sfp_stackoverflow module.
EDIT: Sorry I am an idiot and ran my normal user and the first time I ran as sudo.
@bcoles, do you know of a way to recover data or where its saved? I had a scan running and we lost power and when I brought the system back up, and started spiderfoot my scan data was gone and all my settings were reverted to default again.
@bcoles, do you know of a way to recover data or where its saved? I had a scan running and we lost power and when I brought the system back up, and started spiderfoot my scan data was gone and all my settings were reverted to default again.
Are you absolutely certain you're running SpiderFoot as the same user? The database (which stores scans and settings) is stored in $HOME/.spiderfoot/spiderfoot.db. If you were running SpiderFoot as root with sudo then it will be stored in root's home directory and will not be accessible to other users.
@bcoles, sorry last question until I can get my logs uploaded for my work laptop. When I recovered my instance, the scan says it is still running but I cannot find any proof of that. Should I stop that one and start a new scan or is it actually still running?
Also, thanks for such quick replies!
When I recovered my instance, the scan says it is still running but I cannot find any proof of that. Should I stop that one and start a new scan or is it actually still running?
The scan is toast. The SpiderFoot process was brutally killed while it was running. Under normal circumstances SpiderFoot will update a scan status when the scan is terminated. However, when starved of electricity it has no opportunity to do so.
You can try stopping the scan. You might have to do a dance to trick it. Stop the scan, wait a few seconds (check terminal for output), ctrl+c the process, restart SpiderFoot. If that fails, you can manually update the scan status via the database.
There is minimal harm in leaving the scan in RUNNING state, except that you won't be able to run correlations on it.
No worries. It was a test scan to see if my desktop ran it better than my work laptop. It seems that it does so I may just request permission to run the scan on my personal system. Thanks for the help. I will get the logs off the laptop tomorrow so we can close this.
If you're running SpiderFoot from the GitHub repo, I suggest doing an update. The patch for the sfp_names issue was merged a few hours ago.
So I am unable to find a way to figure out which part of the log belonged to a specific scan. I tried to abort this one probably around the 21st through 23rd. It looks like it could be the last 3 digits of the scan ID in the logs.
I started a new $15 DO box and started a new scan and kept the modules down, I will report back within a couple days to see if it fails.
@smicallef and @bcoles, as expected I started a scan yesterday with the DO box. Pulled up to the terminal this morning and it has been frozen since 02:41UTC log pulled at 14:16UTC, so no traffic for a little under 12 hours. sfp_binstring is not listed either. I redacted any URL and IP address.
tail -n 100 spiderfoot.debug.log
2022-05-03 02:41:05,016 [INFO] sflib.py:229 : Fetched <redacted> (84 bytes in 0.24640202522277832s)
2022-05-03 02:41:05,093 [INFO] sflib.py:229 : Fetched <redacted> (103 bytes in 0.13132190704345703s)
2022-05-03 02:41:05,395 [INFO] sfp_portscan_tcp.py:135 : Spawning thread to check port: 445 on <redacted>
2022-05-03 02:41:05,408 [INFO] sflib.py:229 : Fetched <redacted> (5104 bytes in 0.43798065185546875s)
2022-05-03 02:41:05,428 [INFO] sflib.py:229 : Fetched <redacted> (30 bytes in 0.33182787895202637s)
2022-05-03 02:41:05,441 [INFO] sfp_portscan_tcp.py:135 : Spawning thread to check port: 636 on <redacted>
2022-05-03 02:41:05,446 [INFO] sfp_portscan_tcp.py:135 : Spawning thread to check port: 53 on <redacted>
2022-05-03 02:41:05,460 [INFO] sfp_portscan_tcp.py:135 : Spawning thread to check port: 5631 on <redacted>
2022-05-03 02:41:05,468 [INFO] sfp_portscan_tcp.py:135 : Spawning thread to check port: 123 on <redacted>
2022-05-03 02:41:05,488 [INFO] sfp_portscan_tcp.py:135 : Spawning thread to check port: 443 on <redacted>
2022-05-03 02:41:05,496 [INFO] sfp_portscan_tcp.py:135 : Spawning thread to check port: 161 on <redacted>
2022-05-03 02:41:05,496 [INFO] sfp_portscan_tcp.py:135 : Spawning thread to check port: 5902 on <redacted>
2022-05-03 02:41:05,499 [INFO] sfp_portscan_tcp.py:135 : Spawning thread to check port: 465 on <redacted>
2022-05-03 02:41:05,502 [INFO] sfp_portscan_tcp.py:135 : Spawning thread to check port: 3389 on <redacted>
2022-05-03 02:41:05,671 [INFO] sflib.py:229 : Fetched <redacted> (11021 bytes in 0.23511838912963867s)
2022-05-03 02:41:05,731 [INFO] sflib.py:229 : Fetched <redacted> (1481303 bytes in 0.7137136459350586s)
2022-05-03 02:41:05,797 [INFO] sflib.py:229 : Fetched <redacted> (17652 bytes in 0.37438225746154785s)
2022-05-03 02:41:05,982 [INFO] sflib.py:229 : Fetched <redacted> (221024 bytes in 0.2457866668701172s)
2022-05-03 02:41:06,097 [INFO] sflib.py:229 : Fetched <redacted> (87330 bytes in 0.09899592399597168s)
2022-05-03 02:41:06,112 [INFO] sflib.py:229 : Fetched <redacted> (10257 bytes in 0.43980860710144043s)
2022-05-03 02:41:06,147 [INFO] sflib.py:229 : Fetched <redacted> (13577 bytes in 0.0484316349029541s)
2022-05-03 02:41:06,387 [INFO] sflib.py:229 : Fetched <redacted> (46017 bytes in 0.5878887176513672s)
2022-05-03 02:41:06,767 [INFO] sflib.py:229 : Fetched <redacted> (5263 bytes in 0.6523940563201904s)
2022-05-03 02:41:06,835 [INFO] sflib.py:229 : Fetched <redacted> (6637 bytes in 0.6873178482055664s)
2022-05-03 02:41:06,932 [INFO] sflib.py:229 : Fetched <redacted> (0 bytes in 0.15541720390319824s)
2022-05-03 02:41:07,001 [INFO] sflib.py:229 : Fetched <redacted> (8862 bytes in 0.4812953472137451s)
2022-05-03 02:41:07,083 [INFO] sflib.py:229 : Fetched <redacted> (11323 bytes in 0.24712634086608887s)
2022-05-03 02:41:07,164 [INFO] sflib.py:229 : Fetched <redacted> (17462 bytes in 0.0749669075012207s)
2022-05-03 02:41:07,260 [INFO] sflib.py:229 : Fetched <redacted> (96750 bytes in 0.32541966438293457s)
2022-05-03 02:41:07,473 [INFO] sflib.py:229 : Fetched <redacted> (8610 bytes in 0.3037388324737549s)
2022-05-03 02:41:07,489 [INFO] sflib.py:229 : Fetched <redacted> (177496 bytes in 0.22690534591674805s)
2022-05-03 02:41:07,552 [INFO] sflib.py:229 : Fetched <redacted> (9154 bytes in 0.07722949981689453s)
2022-05-03 02:41:07,747 [INFO] sflib.py:229 : Fetched <redacted> (15615 bytes in 0.7457168102264404s)
2022-05-03 02:41:08,141 [INFO] sflib.py:229 : Fetched <redacted> (38 bytes in 0.37390685081481934s)
2022-05-03 02:41:08,156 [INFO] sflib.py:229 : Fetched <redacted> (396279 bytes in 0.6028487682342529s)
2022-05-03 02:41:08,172 [INFO] sflib.py:229 : Fetched <redacted> (396596 bytes in 0.6740243434906006s)
2022-05-03 02:41:08,286 [INFO] sflib.py:229 : Fetched <redacted> (26937 bytes in 0.13475751876831055s)
2022-05-03 02:41:08,297 [INFO] sflib.py:229 : Fetched <redacted> (44233 bytes in 0.12250375747680664s)
2022-05-03 02:41:08,358 [INFO] sflib.py:229 : Fetched <redacted> (12795 bytes in 0.07054519653320312s)
2022-05-03 02:41:08,440 [INFO] sflib.py:229 : Fetched <redacted> (40782 bytes in 0.08113360404968262s)
2022-05-03 02:41:08,522 [INFO] sflib.py:229 : Fetched <redacted> (10519 bytes in 0.07986927032470703s)
2022-05-03 02:41:08,749 [ERROR] sflib.py:195 : Failed to connect to <redacted>
2022-05-03 02:41:08,751 [DEBUG] sfp_webanalytics.py:62 : Received event, TARGET_WEB_CONTENT, from sfp_spider
2022-05-03 02:41:08,754 [DEBUG] sfp_hashes.py:60 : Received event, TARGET_WEB_CONTENT, from sfp_spider
2022-05-03 02:41:08,768 [DEBUG] threadpool.py:132 : Submitting function "handleEvent" from module "sfp_webanalytics_threadWorker" to thread pool "sharedThreadPool"
2022-05-03 02:41:08,768 [DEBUG] threadpool.py:132 : Submitting function "handleEvent" from module "sfp_subdomain_takeover_threadWorker" to thread pool "sharedThreadPool"
2022-05-03 02:41:08,775 [INFO] sflib.py:229 : Fetched <redacted> (10284 bytes in 0.477541446685791s)
2022-05-03 02:41:08,800 [DEBUG] sfp_ipapico.py:99 : Received event, IP_ADDRESS, from sfp_dnsresolve
2022-05-03 02:41:08,800 [INFO] sflib.py:229 : Fetching (GET): <redacted> (proxy=None, user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0, timeout=5, cookies=None)
2022-05-03 02:41:08,805 [INFO] sflib.py:229 : Fetched <redacted> (504859 bytes in 0.6359727382659912s)
2022-05-03 02:41:08,806 [DEBUG] threadpool.py:132 : Submitting function "handleEvent" from module "sfp_hashes_threadWorker" to thread pool "sharedThreadPool"
2022-05-03 02:41:09,078 [DEBUG] sfp__stor_db.py:60 : Storing an event: HASH
2022-05-03 02:41:09,111 [INFO] sflib.py:229 : Fetched <redacted> (738 bytes in 0.310882568359375s)
2022-05-03 02:41:09,135 [INFO] sflib.py:229 : Fetched <redacted> (16522 bytes in 0.32711195945739746s)
2022-05-03 02:41:09,252 [INFO] sflib.py:229 : Fetched <redacted> (4 bytes in 0.4563143253326416s)
2022-05-03 02:41:09,276 [INFO] sflib.py:229 : Fetched <redacted> (17398 bytes in 0.753760814666748s)
2022-05-03 02:41:09,335 [INFO] sflib.py:229 : Fetched <redacted> (4432 bytes in 0.056966543197631836s)
2022-05-03 02:41:09,366 [INFO] sflib.py:229 : Fetched <redacted> (919 bytes in 0.030499935150146484s)
2022-05-03 02:41:09,414 [INFO] sflib.py:229 : Fetched <redacted> (88083 bytes in 0.16097640991210938s)
2022-05-03 02:41:09,508 [INFO] sflib.py:229 : Fetched <redacted> (23968 bytes in 0.1401965618133545s)
2022-05-03 02:41:10,191 [INFO] sflib.py:229 : Fetched <redacted> (81737 bytes in 1.0508348941802979s)
2022-05-03 02:41:10,461 [INFO] sflib.py:229 : Fetched <redacted> (47878 bytes in 1.0442936420440674s)
2022-05-03 02:41:10,526 [INFO] sflib.py:229 : Fetched <redacted> (12711 bytes in 1.0172665119171143s)
2022-05-03 02:41:10,641 [DEBUG] sfp_hosting.py:94 : Received event, IP_ADDRESS, from sfp_dnsresolve
2022-05-03 02:41:10,644 [DEBUG] threadpool.py:132 : Submitting function "handleEvent" from module "sfp_ipapico_threadWorker" to thread pool "sharedThreadPool"
2022-05-03 02:41:10,670 [DEBUG] sfp_uceprotect.py:139 : Received event, IP_ADDRESS, from sfp_dnsresolve
2022-05-03 02:41:10,670 [DEBUG] sfp_uceprotect.py:106 : Checking UCEPROTECT blacklist: <redacted>
2022-05-03 02:41:10,675 [DEBUG] threadpool.py:132 : Submitting function "handleEvent" from module "sfp_hosting_threadWorker" to thread pool "sharedThreadPool"
2022-05-03 02:41:11,051 [DEBUG] sfp_uceprotect.py:128 : Checking UCEPROTECT blacklist: <redacted>
2022-05-03 02:41:11,072 [INFO] sflib.py:229 : Fetched <redacted> (50265 bytes in 0.5447230339050293s)
2022-05-03 02:41:11,219 [DEBUG] sfp_voipbl.py:164 : Received event, IP_ADDRESS, from sfp_dnsresolve
2022-05-03 02:41:11,220 [DEBUG] sfp_voipbl.py:201 : Checking maliciousness of <redacted> (IP_ADDRESS) with VoIP Blacklist (VoIPBL)
2022-05-03 02:41:11,221 [DEBUG] threadpool.py:132 : Submitting function "handleEvent" from module "sfp_uceprotect_threadWorker" to thread pool "sharedThreadPool"
2022-05-03 02:41:11,261 [INFO] sflib.py:229 : Fetched <redacted> (6280 bytes in 1.0654518604278564s)
2022-05-03 02:41:11,346 [DEBUG] sfp__stor_db.py:60 : Storing an event: GEOINFO
2022-05-03 02:41:11,370 [INFO] sflib.py:229 : Fetched <redacted> (9587 bytes in 0.296353816986084s)
2022-05-03 02:41:11,378 [DEBUG] sfp__stor_db.py:60 : Storing an event: RAW_RIR_DATA
2022-05-03 02:41:12,070 [INFO] sflib.py:229 : Fetched <redacted> (14795 bytes in 1.6065845489501953s)
2022-05-03 02:41:12,124 [DEBUG] sfp__stor_db.py:60 : Storing an event: PROVIDER_HOSTING
2022-05-03 02:41:12,266 [INFO] sflib.py:229 : Fetched <redacted> (24363 bytes in 0.8945517539978027s)
2022-05-03 02:41:12,390 [DEBUG] sfp_emergingthreats.py:134 : Received event, IP_ADDRESS, from sfp_dnsresolve
2022-05-03 02:41:12,390 [DEBUG] sfp_emergingthreats.py:171 : Checking maliciousness of <redacted> with EmergingThreats.net
2022-05-03 02:41:12,392 [DEBUG] threadpool.py:132 : Submitting function "handleEvent" from module "sfp_voipbl_threadWorker" to thread pool "sharedThreadPool"
2022-05-03 02:41:12,394 [INFO] sflib.py:229 : Fetched <redacted> (12230 bytes in 1.123389720916748s)
2022-05-03 02:41:12,400 [INFO] sflib.py:229 : Fetched <redacted> (183443 bytes in 0.13323140144348145s)
2022-05-03 02:41:12,402 [DEBUG] sfp_phishstats.py:128 : Received event, IP_ADDRESS, from sfp_dnsresolve
2022-05-03 02:41:12,402 [INFO] sflib.py:229 : Fetching (GET): <redacted> (proxy=None, user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0, timeout=15, cookies=None)
2022-05-03 02:41:12,405 [DEBUG] threadpool.py:132 : Submitting function "handleEvent" from module "sfp_emergingthreats_threadWorker" to thread pool "sharedThreadPool"
2022-05-03 02:41:12,455 [INFO] sflib.py:229 : Fetched <redacted> (15 bytes in 0.05813956260681152s)
2022-05-03 02:41:12,481 [INFO] sflib.py:229 : Fetched <redacted> (2 bytes in 0.0784149169921875s)
2022-05-03 02:41:12,485 [DEBUG] sfp_botscout.py:139 : Received event, IP_ADDRESS, from sfp_leakix
2022-05-03 02:41:12,486 [INFO] sfp_botscout.py:142 : You enabled sfp_botscout but did not set an API key! Queries will be limited to 100 per day.
2022-05-03 02:41:12,486 [INFO] sflib.py:229 : Fetching (GET): <redacted> (proxy=None, user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0, timeout=5, cookies=None)
2022-05-03 02:41:12,492 [DEBUG] threadpool.py:132 : Submitting function "handleEvent" from module "sfp_phishstats_threadWorker" to thread pool "sharedThreadPool"
2022-05-03 02:41:12,549 [INFO] sflib.py:229 : Fetched <redacted> (65917 bytes in 0.14115405082702637s)
2022-05-03 02:41:12,570 [INFO] sflib.py:229 : Fetched <redacted> (4972 bytes in 0.1144254207611084s)
2022-05-03 02:41:19,032 [INFO] sflib.py:229 : Fetched <redacted> (13675 bytes in 6.940148591995239s)
2022-05-03 02:41:19,043 [INFO] sflib.py:229 : Fetched <redacted> (6 bytes in 6.557460784912109s)
2022-05-03 02:41:19,054 [DEBUG] sfp_torexits.py:176 : Received event, AFFILIATE_IPADDR, from sfp_dnsneighbor
2022-05-03 02:41:19,063 [DEBUG] threadpool.py:132 : Submitting function "handleEvent" from module "sfp_botscout_threadWorker" to thread pool "sharedThreadPool"
root@ubuntu-s-2vcpu-2gb-nyc1-01-chrome:~/.spiderfoot/logs# date
Tue May 3 14:16:33 UTC 2022
How many modules and threads were enabled approximately?
Which version of Python?
Are there errors in spiderfoot.error.log ? Are there OOM error in dmesg ?
None of these modules have known issues.
2022-05-03 02:41:08,768 [DEBUG] threadpool.py:132 : Submitting function "handleEvent" from module "sfp_webanalytics_threadWorker" to thread pool "sharedThreadPool"
2022-05-03 02:41:08,768 [DEBUG] threadpool.py:132 : Submitting function "handleEvent" from module "sfp_subdomain_takeover_threadWorker" to thread pool "sharedThreadPool"
2022-05-03 02:41:08,806 [DEBUG] threadpool.py:132 : Submitting function "handleEvent" from module "sfp_hashes_threadWorker" to thread pool "sharedThreadPool"
2022-05-03 02:41:10,644 [DEBUG] threadpool.py:132 : Submitting function "handleEvent" from module "sfp_ipapico_threadWorker" to thread pool "sharedThreadPool"
2022-05-03 02:41:10,675 [DEBUG] threadpool.py:132 : Submitting function "handleEvent" from module "sfp_hosting_threadWorker" to thread pool "sharedThreadPool"
2022-05-03 02:41:11,221 [DEBUG] threadpool.py:132 : Submitting function "handleEvent" from module "sfp_uceprotect_threadWorker" to thread pool "sharedThreadPool"
2022-05-03 02:41:12,392 [DEBUG] threadpool.py:132 : Submitting function "handleEvent" from module "sfp_voipbl_threadWorker" to thread pool "sharedThreadPool"
2022-05-03 02:41:12,405 [DEBUG] threadpool.py:132 : Submitting function "handleEvent" from module "sfp_emergingthreats_threadWorker" to thread pool "sharedThreadPool"
2022-05-03 02:41:12,492 [DEBUG] threadpool.py:132 : Submitting function "handleEvent" from module "sfp_phishstats_threadWorker" to thread pool "sharedThreadPool"
2022-05-03 02:41:19,063 [DEBUG] threadpool.py:132 : Submitting function "handleEvent" from module "sfp_botscout_threadWorker" to thread pool "sharedThreadPool"
I ran my usual test scan with these modules included (although some without an API key configured) and wasn't able to reproduce the issue. But that doesn't mean much.
How many modules and threads were enabled approximately? Scan was setup with the default 3 modules. I do not know hoe many threads were going. It still hasn't recovered yet.
Which version of Python? Python 3.8.10
Are there errors in spiderfoot.error.log ? Are there OOM error in dmesg ? Only errors I see in spiderfoot.error.log are unable to fetch errors. There is no dmesg log just debug and error in the logs folder.