PyFunceble
PyFunceble copied to clipboard
BUG: TypeError(f"<data> should be {dict}, {type(data)} given.")
Description
Process pyfunceble_producer_worker_1:
Traceback (most recent call last):
File "/usr/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/cli/processes/workers/base.py", line 395, in run
raise exception
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/cli/processes/workers/base.py", line 373, in run
result = self.target(consumed)
^^^^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/cli/processes/workers/producer.py", line 406, in target
self.collection_query_tool.push(test_result)
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/query/collection.py", line 393, in wrapper
return func(self, *args, **kwargs) # pylint: disable=not-callable
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/query/collection.py", line 511, in push
self.__push_whois(checker_status)
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/query/collection.py", line 626, in __push_whois
raise TypeError(f"<data> should be {dict}, {type(data)} given.")
TypeError: <data> should be <class 'dict'>, <class 'PyFunceble.checker.availability.status.AvailabilityCheckerStatus'> given.
Configuration
cli_testing:
autocontinue: true
cidr_expand: false
inactive-db: true
db_type: mariadb
display_mode:
all: true
colour: true
dots: false
execution_time: true
less: false
percentage: true
quiet: false
simple: false
status: all
file_generation:
analytic: true
hosts: false
merge_output_dirs: false
no_file: false
plain: true
unified_results: false
local_network: false
sorting_mode:
hierarchical: true
days_between:
db_clean: 14
db_retest: 7
debug:
active: true
level: critical
lookup:
collection: true
dns: true
http_status_code: true
netinfo: true
reputation: false
special: true
timeout: 5
whois: true
user_agent:
browser: firefox
custom: null
platform: linux
dns:
follow_server_order: false
trust_server: true
protocol: UDP
server:
- 127.0.0.1:5302
- 9.9.9.10
- 149.112.112.10
collection:
preferred_status_origin: latest
push: true
url_base: https://collection.dead-hosts.funilrys.com
max_http_retries: 1
proxy:
# global:
# http: socks5h://127.0.0.1:9050
# https: socks5h://127.0.0.1:9050
rules:
- http: socks5h://127.0.0.1:9050
https: socks5h://127.0.0.1:9050
tld:
- onion
Reproduction
-
pyfunceble -f https://gist.githubusercontent.com/okutbay/5b4974b70673dfdcc21c517632c1f984/raw/5bbdaba2927a5a5e751243daa251a9d242f3de0b/free_email_provider_domains.txt --local --dns-protocol HTTPS --dns "https://dns10.quad9.net/dns-query","https://anycast.censurfridns.dk/dns-query","https://unicast.censurfridns.dk/dns-query"
Expected behavior
Screenshots
Versions
OS: ubu 22.04
Python Version: 3.11
PyFunceble Version:
pyfunceble --version
^CTraceback (most recent call last):
File "/home/$USER/.local/bin/pyfunceble", line 8, in <module>
sys.exit(tool())
^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/cli/entry_points/pyfunceble/cli.py", line 1237, in tool
PyFunceble.facility.ConfigLoader.start()
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/config/loader.py", line 405, in start
config = self.get_config_file_content()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/config/loader.py", line 334, in get_config_file_content
self.download_dynamic_infrastructure_files()
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/config/loader.py", line 315, in download_dynamic_infrastructure_files
UserAgentsDownloader().start()
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/downloader/base.py", line 285, in start
if DownloadHelper(self.download_link).download_text(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/helpers/download.py", line 235, in download_text
req = session.get(self.url, verify=self.certificate_validation)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/requests/sessions.py", line 602, in get
return self.request("GET", url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 790, in urlopen
response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 536, in _make_request
response = conn.getresponse()
^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/urllib3/connection.py", line 461, in getresponse
httplib_response = super().getresponse()
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/http/client.py", line 1374, in getresponse
response.begin()
File "/usr/lib/python3.11/http/client.py", line 318, in begin
version, status, reason = self._read_status()
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/http/client.py", line 279, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/socket.py", line 705, in readinto
return self._sock.recv_into(b)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/ssl.py", line 1278, in recv_into
return self.read(nbytes, buffer)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/ssl.py", line 1134, in read
return self._sslobj.read(len, buffer)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
Additional context
Frustration... I've swiched to CSV to avoif the dict issue, only to get this...
Process pyfunceble_producer_worker_1:
Traceback (most recent call last):
File "/usr/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/cli/processes/workers/base.py", line 395, in run
raise exception
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/cli/processes/workers/base.py", line 373, in run
result = self.target(consumed)
^^^^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/cli/processes/workers/producer.py", line 406, in target
self.collection_query_tool.push(test_result)
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/query/collection.py", line 393, in wrapper
return func(self, *args, **kwargs) # pylint: disable=not-callable
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/query/collection.py", line 511, in push
self.__push_whois(checker_status)
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/query/collection.py", line 626, in __push_whois
raise TypeError(f"<data> should be {dict}, {type(data)} given.")
TypeError: <data> should be <class 'dict'>, <class 'PyFunceble.checker.availability.status.AvailabilityCheckerStatus'> given.
Wondering who wrote these lines years ago... don't you blame me.... :smile: :smirk:
https://github.com/funilrys/PyFunceble/blob/dev/docs/facts/known_issues.rst#sql-missing-default-data-in-whois-table
https://github.com/funilrys/PyFunceble/blob/dev/docs/facts/known_issues.rst?plain=1#L71-L104
There is a thing or two, that is confusing...
SELECT COUNT(*) FROM pyfunceble_whois_record
WHERE id IS NOT NULL;
returns 58595
records (from a clean "installed" db) So which domain is causing this issue, that is missing from the log. Meaning already building the WHOIS db from scratch?
- Why did running the known workaround NOT solve this?
- Going to run with
--logging-level debug
- Will return with, hopefully, more knowledge
Unlike what I was believing, this issue is not related to WRITE data, but to READ data...
This is still a issue in 4.2.7
Screenshot
pyfunceble --output-location ./ -f ad_servers.txt emd.txt exp.txt fsa.txt grm.txt hjk.txt mmt.txt pha.txt psh.txt pup.txt
██████╗ ██╗ ██╗███████╗██╗ ██╗███╗ ██╗ ██████╗███████╗██████╗ ██╗ ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║ ██║████╗ ██║██╔════╝██╔════╝██╔══██╗██║ ██╔════╝
██████╔╝ ╚████╔╝ █████╗ ██║ ██║██╔██╗ ██║██║ █████╗ ██████╔╝██║ █████╗
██╔═══╝ ╚██╔╝ ██╔══╝ ██║ ██║██║╚██╗██║██║ ██╔══╝ ██╔══██╗██║ ██╔══╝
██║ ██║ ██║ ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝
Subject Status Source Expiration Date Registrar HTTP Code Checker
---------------------------------------------------------------------------------------------------- ----------- ---------- ----------------- ------------------------------ ---------- -------------
005.free-counter.co.uk ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY
006.free-counter.co.uk ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY
007.free-counter.co.uk ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY
04fd-74b3-b2f6-c645.reporo.net ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY
008.free-counter.co.uk ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY
06c5-dbbd-eb79-4cd4.reporo.net ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY
060810131024.c.mystat-in.net ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY
02gzx.voluumtrk.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY
007.go2cloud.org ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY
011i5.voluumtrk.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY
0602v.voluumtrk.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY
05ni7.voluumtrk.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY
06rph.voluumtrk.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY
0075-7112-e7eb-f9b9.reporo.net ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY
08.185.87.08.liveadvert.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY
08.185.87.05.liveadvert.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY
08.185.87.100.liveadvert.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY
08.185.87.09.liveadvert.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY
08.185.87.03.liveadvert.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY
08.185.87.1.liveadvert.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY
08.185.87.04.liveadvert.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY
050003.voodoo.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY
08.185.87.06.liveadvert.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY
08.185.87.10.liveadvert.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY
08.185.87.101.liveadvert.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY
08.185.87.07.liveadvert.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY
08.185.87.103.liveadvert.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY
08.185.87.104.liveadvert.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY
0124498474f7c13ac9a2-6b191446002b31342189d56cabcf5227.r11.cf2.rackcdn.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY
050005.voodoo.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY
Process pyfunceble_producer_worker_1:
Traceback (most recent call last):
File "/usr/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/home/spirillen/.local/lib/python3.11/site-packages/PyFunceble/cli/processes/workers/base.py", line 395, in run
raise exception
File "/home/spirillen/.local/lib/python3.11/site-packages/PyFunceble/cli/processes/workers/base.py", line 373, in run
result = self.target(consumed)
^^^^^^^^^^^^^^^^^^^^^
File "/home/spirillen/.local/lib/python3.11/site-packages/PyFunceble/cli/processes/workers/producer.py", line 406, in target
self.collection_query_tool.push(test_result)
File "/home/spirillen/.local/lib/python3.11/site-packages/PyFunceble/query/collection.py", line 393, in wrapper
return func(self, *args, **kwargs) # pylint: disable=not-callable
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/spirillen/.local/lib/python3.11/site-packages/PyFunceble/query/collection.py", line 511, in push
self.__push_whois(checker_status)
File "/home/spirillen/.local/lib/python3.11/site-packages/PyFunceble/query/collection.py", line 626, in __push_whois
raise TypeError(f"<data> should be {dict}, {type(data)} given.")
TypeError: <data> should be <class 'dict'>, <class 'PyFunceble.checker.availability.status.AvailabilityCheckerStatus'> given.
^C
Thank you for NOT using PyFunceble!
When can We expect it to be fixed or see the rust release?