krawler
krawler copied to clipboard
brianmadden
A web crawling framework written in Kotlin
In url pages like http://bvmf.bmfbovespa.com.br/pt-br/mercados/acoes/empresas/ExecutaAcaoConsultaInfoEmp.asp?CodCVM=25062&ViewDoc=1&AnoDoc=2021&VersaoDoc=2&NumSeqDoc=99821#a the response are returning without utf-8 format like: Consumo CÃclico / Construção Civil / Incorporações and should come something like Consumo Cíclico / Construção Civil...
The example project page (https://github.com/brianmadden/krawler/blob/master/...) in README return 404 error.
I've got of every unsafe dereference `!!`. I hope the default case in RoboMinder is ok.
https://github.com/brianmadden/krawler/blob/master/src/main/kotlin/io/thelandscape/krawler/crawler/Krawler.kt#L231 and https://github.com/brianmadden/krawler/blob/master/src/main/kotlin/io/thelandscape/krawler/crawler/Krawler.kt#L219 Should be modified to remove root page mappings when any queue entries associated with them are removed from the queue.
