CrawlerX
CrawlerX copied to clipboard
fix encoding setting invalid
how to reproduce:
the page encoding is gb2312,the code is like:
crawler.request("pageUrl",{decode:"gb2312",deep:3,concurrency:4}, function(err, $, body,url,resp){
$.spider(); //start recursive crawl
});
when the page include Chinese, then I got garbled body content.
what i do:
The code in line 108 is
q.push({deep:deep,url:url,options:options},getCallback(deep) );
The encoding of options is lost, so I change it to
q.push({deep:deep,url:url,options:options,encoding:options.encoding},getCallback(deep) );