gospider icon indicating copy to clipboard operation
gospider copied to clipboard

Update crawler.go

Open ghsec opened this issue 3 years ago • 0 comments

when crawling some sites output is like

js/image
text/js
text/html
text/xml|js
js/js/js/
text/javascript
text/plain
e.t.c

the reason is jslinkfinder module. easy way fix this add to blacklist which is default true

or add this mime type on blacklists and don't add to in output URLs https://github.com/fuzzdb-project/fuzzdb/blob/master/attack/mimetypes/MimeTypes.txt

ghsec avatar Oct 30 '21 20:10 ghsec