privacyscanner

[jayvdb]

Type: enhancement

https://pypi.org/project/privacyscanner/ is a relatively new tools, a little over a year old, which is now quite stable and very useful.

It is however a bit slow. e.g. 4 mins

> privacyscanner scan --scan-modules chromedevtools,dns,testssl_https https://blog.usni.org > /dev/null
chromedevtools> 2020-02-21 11:30:28,178: [INFO] Starting chromedevtools (scanner.py:180)
chromedevtools> 2020-02-21 11:30:56,691: [INFO] Finished chromedevtools (scanner.py:204)
dns> 2020-02-21 11:30:56,724: [INFO] Starting dns (scanner.py:180)
dns> 2020-02-21 11:30:56,738: [WARNING] GeoIP database not available. Country lookup disabled. (dns.py:173)
dns> 2020-02-21 11:30:56,743: [WARNING] [] (dns.py:49)
dns> 2020-02-21 11:30:56,767: [INFO] Finished dns (scanner.py:204)
testssl_https> 2020-02-21 11:30:56,768: [INFO] Starting testssl_https (scanner.py:180)
testssl_https> 2020-02-21 11:30:56,772: [INFO] Current stage: basic (base.py:66)
testssl_https> 2020-02-21 11:32:52,369: [INFO] testssl.sh result is complete. (base.py:97)
testssl_https> 2020-02-21 11:32:52,370: [INFO] Next stage: vulns (base.py:119)
testssl_https> 2020-02-21 11:32:52,385: [INFO] Finished testssl_https (scanner.py:204)
testssl_https> 2020-02-21 11:33:02,398: [INFO] Starting testssl_https (scanner.py:180)
testssl_https> 2020-02-21 11:33:02,407: [INFO] Current stage: vulns (base.py:66)
testssl_https> 2020-02-21 11:33:34,014: [INFO] testssl.sh result is complete. (base.py:97)
testssl_https> 2020-02-21 11:33:34,015: [INFO] Next stage: vulns_ids (base.py:119)
testssl_https> 2020-02-21 11:33:34,035: [INFO] Finished testssl_https (scanner.py:204)
testssl_https> 2020-02-21 11:33:44,048: [INFO] Starting testssl_https (scanner.py:180)
testssl_https> 2020-02-21 11:33:44,054: [INFO] Current stage: vulns_ids (base.py:66)
testssl_https> 2020-02-21 11:34:44,907: [INFO] testssl.sh result is complete. (base.py:97)
testssl_https> 2020-02-21 11:34:44,908: [INFO] vulns_ids was the final stage. (base.py:117)
testssl_https> 2020-02-21 11:34:44,928: [INFO] Finished testssl_https (scanner.py:204)

The result of that is a very detailed json file.

It includes several mixed-content checks (part of chromedevtools, which is ~30s), which I think would be helpful for https://github.com/EFForg/https-everywhere/issues/18491 I haven't yet played with this extensively to verify it detects most common mixed/insecure media checks, but even if it only detects some, that is a win.

It's ssl module uses https://github.com/drwetter/testssl.sh under the covers.

Running just the chromedevtools scanner in CI on the 'shortest target(s)', i.e. domain level targets, and storing the results in the repo, would help identify any significant changes in the targets, such as buyouts, non-functional sites showing errors, etc.

It also has an option to import previous results, but I ran into a bug with that https://github.com/PrivacyScore/privacyscanner/issues/26