news-crawler icon indicating copy to clipboard operation
news-crawler copied to clipboard

What if there isn't any news after 2017.03?

Open YunBAI-PSL opened this issue 2 years ago • 4 comments

Dear Author,

Thanks for your nice job. I run your codes and find there isn't news after 2017.03. But I need some recent news, how do you handle this kind of problem?

Many thanks.

YunBAI-PSL avatar Mar 04 '22 10:03 YunBAI-PSL

Hi, thank you for your interests.

Did you change the time range setting in the settings/*.cfg files? Also, you may also need to set a larger sleep time because frequent visits to nytimes from the same IP may trigger their reCAPTCHA verification.

LuChang-CS avatar Mar 04 '22 20:03 LuChang-CS

Even if you set the date in cfg, data cannot be crawled after 2017.

swthinking avatar Apr 15 '22 05:04 swthinking

Maybe name class has changed, so you can not get all link paper. You can check line 31

ducnva avatar Apr 06 '23 05:04 ducnva

Just change line 31 to elements = soup.table.find_all('a') .

Just test, it runs without problem.

liyucheng09 avatar Apr 11 '23 14:04 liyucheng09