PyFunceble
PyFunceble copied to clipboard
funilrys
HELP: Extremly poor and slow testing
Description
After running this test pyfunceble --dns-lookup-only --dns 192.168.1.54:5302 --database-type csv -f https://raw.githubusercontent.com/blocklistproject/Lists/master/porn.txt for a week I have only reached to boystreaking.blogspot.fr
wc -l /tmp/pyfunceble/output/porn.txt/domains/INACTIVE/list /tmp/pyfunceble/output/porn.txt/domains/ACTIVE/list
399957 /tmp/pyfunceble/output/porn.txt/domains/INACTIVE/list
44642 /tmp/pyfunceble/output/porn.txt/domains/ACTIVE/list
444599 total
cat ~/.config/PyFunceble/.PyFunceble.overwrite.yaml
cli_testing:
cidr_expand: true
db_type: mariadb
display_mode:
all: true
colour: true
dots: false
execution_time: true
less: false
percentage: true
quiet: false
simple: false
status: all
file_generation:
analytic: true
hosts: false
merge_output_dirs: false
no_file: false
plain: true
unified_results: false
local_network: true
sorting_mode:
hierarchical: true
debug:
active: false
level: critical
lookup:
dns: true
http_status_code: true
netinfo: true
reputation: false
special: true
timeout: 5
whois: true
user_agent:
browser: firefox
custom: null
platform: linux
dns:
follow_server_order: false
trust_server: false
protocol: UDP
server:
- 130.226.161.34
- 89.233.43.71
- 185.121.177.177
- 163.172.168.171
- 192.168.1.100:5302
- 192.168.1.6
- 9.9.9.10
- 149.112.112.10
collection:
push: true
url_base: https://collection.dead-hosts.funilrys.com
cat /etc/fstab
none /tmp/pyfunceble/ ramfs noauto,user,mode=0777 0 0
df -h /tmp/pyfunceble/
Filesystem Size Used Avail Use% Mounted on
none 0 0 0 - /tmp/pyfunceble
How come it takes a week to test against 1 DNS server.... and how to optimize this
Even if this should be extremely slow, the absolutely max time for testing each record is 1400ms (recursor lookup time if connection is temporary slow)
If you choose to set trust_server: false, PyFunceble will go through all of the server before setting the status from the DNS Lookup.
CF: https://pyfunceble.readthedocs.io/en/dev/configuration/index.html#dns-trust-server
Description: Activates or disables the trust mode. When the trust mode is active and the first read DNS server gives us a negative response (without any error), we take it as it is.
Otherwise, when the trust mode is disabled, when the first read DNS server gives us a negative response (without any error), we still ask all other DNS servers that were given or found.
trust_server:
Your right... forgot to set that one in my command line.... thanks for spotting this one
Another question now you are online
About the --soometest-only if I add IE --dns-lookup-only --http-only then it will test both and yet only for DNS and HTTP status code or? that's not something I have lookup, but I don't recall reading about this in the merge you made back then
I still think it is very slow with the CSV.... while for some reason the SQL have speed up :hand:
https://asciinema.org/a/tAT4MGPfu0vwxetpdlnNKpqQr
That's the reason SQL databases exists ... CSV needs to be loaded in memory .... When SQL is mostly delegated to a optimized system and format.