CrawlerX icon indicating copy to clipboard operation
CrawlerX copied to clipboard

fix encoding setting invalid

Open ilife5 opened this issue 9 years ago • 0 comments

how to reproduce:

the page encoding is gb2312,the code is like:

crawler.request("pageUrl",{decode:"gb2312",deep:3,concurrency:4}, function(err, $, body,url,resp){
    $.spider();  //start recursive crawl
});

when the page include Chinese, then I got garbled body content.

what i do:

The code in line 108 is

q.push({deep:deep,url:url,options:options},getCallback(deep) );

The encoding of options is lost, so I change it to

q.push({deep:deep,url:url,options:options,encoding:options.encoding},getCallback(deep) );

ilife5 avatar May 28 '15 03:05 ilife5