sherlock-project
Reduce false positives (those not caused by WAFs or bot detection)
The following targets were fixed or partially fixed: ~~Archive[.]org~~ (eh. sometimes. it has a loading problem that causes other F+s... some were fixed) Archive of Our Own CGTrader CNET Contently Eintracht Frankfurt Forum GeeksForGeeks Genius[.]com Artists Genius[.]com Users Gumroad HackerNews HackerRank IFTTT Kongregate Linktree OpenStreetMap Pinkbike Polymart (often hits bot detection, but can be bypassed with some proxies) Slides Splits[.]io Strava Telegram xHamster YandexMusic eintracht jeuxvideo
The following targets were removed: BitcoinForum (likely defunct) G2G HexRPG (auth wall) Metacritic ModelHub (defunct) Oracle Communities (auth wall)
Misc changes: Default User Agent updated
ModelHub was not added to ./removed_sites.[md|json] as the platform itself is confirmed to be shutting down (and will therefore never return to Sherlock). The other removed targets were documented normally.
Fiverr, Euw, NationStates Nations, NationStates Regions, and a couple others remain occasionally problematic as they are behind WAFs and bot detection services. These WAF-induced false positives are resolved with sister PR #2069 and have partial support for a decent proxy being discussed in #2081.
Issues: Fixes #904 Fixes #1966 Fixes #1999 (with sister PR #2069) Fixes #2027 (with sister PR #2069) Fixes #2071
Collateral (trumped or negated): Closes #1843 // Removes jeuxvideo instead of applying a fix (a fix is being applied here). Closes #2083 // Removes ModelHub, which is an action taken here. Would be negated by merge. Closes #2096 // Fixes Archive[.]org, which is an action taken here. Would be negated by merge.
A lil bit larger of a pr than first expected........
I think that's all of em, though... Well, I hope that's all of em When combined with #2069, I'm no longer experiencing any false positives in my results. I'm sure there's some username combo that causes one, but none that I've stumbled upon
For general testing purposes, ppfeister:rc/combopoc currently reflects master with this, #2069, #2070, and #2092 all merged.
(ppfeister:rc/combopoc itself has a messy merge history though -- don't use that branch for any merging upstream)
Cheers.
Thank you so much @ppfeister for fixing all of this, I really appriciate it!
