Soft 404 detection

[dogancanbakir]

Please describe your feature request and the use case of this feature:

A soft 404 happens when a web server responds with a 200 OK HTTP response code for a page that doesn't exist rather than the appropriate 404 Not Found. Thus, soft 404s can limit or slow down a target's crawl coverage because of the time and power spent crawling these identical URLs instead of pages with unique content.

Consequently, detecting and skipping these pages will possibly increase performance and coverage.

Resources:

https://github.com/dogancanbakir/soft-404