Running SpiderFoot Plugins Detached from UI

[iklobato]

Description:

Moreover, I am working to evaluate a custom functionality to run SpiderFoot plugins detached from the user interface, but by using SpiderFoot internal classes. One of the key motivation for this work was to eliminate the requirement of starting the SpiderFoot separate server, including all the plugins that needs to be tested. I replaced the SQLite backend database with MongoDB to explore its compatibility with SpiderFoot. Note that I only used MongoDB for experimental purposes only.

Context:

My primary inquiry centers on developing a method to sequentially activate every SpiderFoot module in an autonomous script devoid of server reliance. I aspire to refine the execution flow and boost efficiency by seamlessly chaining each module's operation through single-run script invocation.

Question

The script below attempts this by initializing all modules, yet some opaque failure subsequently arises, eluding detection even with logging at the most granular level. A deeper examination is warranted to unravel the underlying dysfunction and thereby fulfill the objective of a self-contained, server-independent SpiderFoot analysis suite. How can I get the modules to run independently of the server?

Script used:

The Python script code snippet below shows the creation of an environment and execution:

import logging
import os
from pymongo.errors import ServerSelectionTimeoutError
from sf import sfConfig
from sflib import SpiderFoot
from sfscan import startSpiderFootScanner
from spiderfoot import SpiderFootHelpers, SpiderFootCorrelator, SpiderFootDb

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(message)s",
    handlers=[logging.StreamHandler()]
)


class SpiderFootAppCustom:
    def __init__(self, _sfConfig, _sfModules, _sfWebUiConfig):
        self.sfConfig = _sfConfig
        self.sfModules = _sfModules
        self.sfWebUiConfig = _sfWebUiConfig
        self.dbh = SpiderFootDb(self.sfConfig, init=True)
        correlations_dir = os.path.dirname(os.path.abspath(__file__)) + '/correlations/'
        correlationRulesRaw = SpiderFootHelpers.loadCorrelationRulesRaw(correlations_dir, ['template.yaml'])
        correlator = SpiderFootCorrelator(self.dbh, correlationRulesRaw)
        sfConfig['__correlationrules__'] = correlator.get_ruleset()
        logging.info("SpiderFootApp initialized")

    def start_scan(self):
        logging.info("Starting scan")
        target = "iktech.solutions"
        targetType = SpiderFootHelpers.targetTypeFromString(target)
        cfg = self.prepare_config()
        self.run_scan(target, targetType, self.sfModules, cfg)

    def modules_as_list(self):
        logging.info("Preparing module list")
        return list(self.sfModules.keys())

    def prepare_config(self):
        logging.info("Preparing config")
        cfg = self.sfConfig.copy()
        cfg['_debug'] = True
        return cfg

    def run_scan(self, target, targetType, modlist, cfg):
        logging.info("Running scan")
        scanName = target
        scanId = SpiderFootHelpers.genScanInstanceId()
        try:
            startSpiderFootScanner(None, scanName, scanId, target, targetType, self.modules_as_list(), cfg)
        except ServerSelectionTimeoutError as e:
            self.log_error(f"Error at Database connection: {e}")
            return
        scan_status = None
        while True:
            info = self.dbh.scanInstanceGet(scanId)
            info = info[0]
            scan_status = info['status']
            if scan_status in ['RUNNING', 'PENDING']:
                logging.debug(f"Scan status: {scan_status}")
                continue
            if scan_status in ["ERROR-FAILED", "ABORT-REQUESTED", "ABORTED"]:
                logging.critical(f"Scan status: {scan_status}")
                break
        self.log_info(f"Scan completed with status {scan_status}")

    def log_error(self, message):
        logging.error(message)

    def log_info(self, message):
        logging.info(message)


if __name__ == "__main__":
    mod_dir = os.path.dirname(os.path.abspath(__file__)) + '/modules/'
    sfModules = SpiderFootHelpers.loadModulesAsDict(mod_dir, ['sfp_template.py'])
    sfConfig['__modules__'] = sfModules
    sf = SpiderFoot(sfConfig)
    sfWebUiConfig = {}
    app = SpiderFootAppCustom(sfConfig, sfModules, sfWebUiConfig)
    app.start_scan()

Feedback requested

I welcome any insights, suggestions, or guidelines regarding SpiderFoot modules being separated from the UI during script usage. Additionally, any suggestions on how to improve the quality of the text or improve the testing process will be greatly appreciated. I would also like to say thank you for the wonderful osint tool.