gospider
gospider copied to clipboard
Update crawler.go
when crawling some sites output is like
js/image
text/js
text/html
text/xml|js
js/js/js/
text/javascript
text/plain
e.t.c
the reason is jslinkfinder module. easy way fix this add to blacklist which is default true
or add this mime type on blacklists and don't add to in output URLs https://github.com/fuzzdb-project/fuzzdb/blob/master/attack/mimetypes/MimeTypes.txt