FEATURE: Reputation filter using the proxy connection

[spirillen]

Description

If you enables --reputation-lookup && --reputation then they do not obey the proxy settings

File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='reputation.alienvault.com', port=443): Max retries exceeded with url: /reputation.data (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fec793377d0>: Failed to establish a new connection: [Errno -2] Name or service not known'))

Log

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/home/joakim/.local/lib/python3.11/site-packages/PyFunceble/cli/processes/workers/base.py", line 395, in run
raise exception
File "/home/joakim/.local/lib/python3.11/site-packages/PyFunceble/cli/processes/workers/base.py", line 373, in run
result = self.target(consumed)
^^^^^^^^^^^^^^^^^^^^^
File "/home/joakim/.local/lib/python3.11/site-packages/PyFunceble/cli/processes/workers/tester.py", line 292, in target
.query_status()
^^^^^^^^^^^^^^
File "/home/joakim/.local/lib/python3.11/site-packages/PyFunceble/checker/base.py", line 161, in wrapper
return func(self, *args, **kwargs)  # pylint: disable=not-callable
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/joakim/.local/lib/python3.11/site-packages/PyFunceble/checker/base.py", line 186, in wrapper
result = func(self, *args, **kwargs)  # pylint: disable=not-callable
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/joakim/.local/lib/python3.11/site-packages/PyFunceble/checker/reputation/domain_and_ip.py", line 95, in query_status
result = query_object.query_status()
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/joakim/.local/lib/python3.11/site-packages/PyFunceble/checker/base.py", line 161, in wrapper
return func(self, *args, **kwargs)  # pylint: disable=not-callable
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/joakim/.local/lib/python3.11/site-packages/PyFunceble/checker/base.py", line 186, in wrapper
result = func(self, *args, **kwargs)  # pylint: disable=not-callable
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/joakim/.local/lib/python3.11/site-packages/PyFunceble/checker/reputation/base.py", line 309, in query_status
self.try_to_query_status_from_dns_lookup()
File "/home/joakim/.local/lib/python3.11/site-packages/PyFunceble/checker/reputation/base.py", line 198, in try_to_query_status_from_dns_lookup
if subject in self.ipv4_reputation_query_tool:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/joakim/.local/lib/python3.11/site-packages/PyFunceble/dataset/ipv4_reputation.py", line 77, in __contains__
with self.get_content() as file_stream:
^^^^^^^^^^^^^^^^^^
File "/home/joakim/.local/lib/python3.11/site-packages/PyFunceble/dataset/base.py", line 109, in wrapper
return func(self, *args, **kwargs)  # pylint: disable=not-callable
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/joakim/.local/lib/python3.11/site-packages/PyFunceble/dataset/ipv4_reputation.py", line 100, in get_content
self.DOWNLOADER.start()
File "/home/joakim/.local/lib/python3.11/site-packages/PyFunceble/downloader/base.py", line 285, in start
if DownloadHelper(self.download_link).download_text(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/joakim/.local/lib/python3.11/site-packages/PyFunceble/helpers/download.py", line 235, in download_text
req = session.get(self.url, verify=self.certificate_validation)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 557, in get
return self.request('GET', url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 544, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 657, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='reputation.alienvault.com', port=443): Max retries exceeded with url: /reputation.data (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbb2b7d3990>: Failed to establish a new connection: [Errno -2] Name or service not known'))

Possible Solution

make sure the global[rules] are used if set.

proxy:
  global:
    http: socks5h://127.0.0.1:9050
    https: socks5h://127.0.0.1:9050
  rules:
    - http: socks5h://127.0.0.1:9050
      https: socks5h://127.0.0.1:9050
      tld:
        - onion

Considered Alternative

rather boring as I would have to force the entire box to use proxy ala

alias proxyhttp="export http_proxy='http://$HOST:$PORT';export https_proxy='http://$HOST:$PORT'" # Squid
alias proxytor="export http_proxy='socks5h://localhost:9050';export https_proxy='socks5h://localhost:9050'" # Tor
alias proxyoff="unset http_proxy; unset https_proxy"

Additional context

Just stumbled on it

[spirillen]

Thought:

In addition, optional use assigned --dns, but should never be a default but a argument only

[funilrys]

Just a status update on this: In the isuse-349 branch, a patch has been written.

Although it may work at runtime - at least on my machine, it doesn't pass the CI/CD tests because of circular import. So, I'll have to fix that first.

[spirillen]

So, I'll have to fix that first.

... in rust :smile: As said, just stumbled on it as I was bored, not something I use. So guessing it isn't that important as nobody else has stumbled on it.