Add feature to exclude the same http response

[0xAwali]

trafficstars

while scanning wide scope there are a lot of urls have the same base code so they have the same http response so it's time to let httpx exclude these urls based on comparing every http response with others

[ibitebyt3s]

The anew tool might be a good fit for this only as a standalone program.

Wanted to suggest using its code in this project at first but I can't remember 100% if this was true since it happend about 2 years ago but anyway.

When using httpx with the -tech-detect option there will be times when a technology isn't detected in the same host, thus making anew useless in this case.

Lets say that in the first run it detected [php, Wordpress] and in the second run it detected only [php]. Using anew to check for duplicates will still keep duplicate hosts if their technologies detected are diffrent. Same could happen with other CLI options like -ip

[ehsandeep]

More ideas - https://twitter.com/har1sec/status/1600445181115109377

when -path is used, make a request to / to get number of lines and words and use this as a baseline to filter response (quick custom 404 filter)