fact-bounty icon indicating copy to clipboard operation
fact-bounty copied to clipboard

Published 20 hours ago • verified publisher icon scorelab

Add tests for Scrapy Crawlers

Open Anmolbansal1 opened this issue 5 years ago • 6 comments

Expected Behavior

There should be a test method to test all scrapy crawlers for -

  • consistent output across all crawlers
  • proper hyperlinks in pages
  • all content is extracted from news page
  • they should be lite such that they can be enforced with travis ci

Current Behavior

Currently there are no tests.

Context

Tests makes the coding process more Agile. When more and more crawlers are added to scrapy project, sometimes structure needs to be changed. However, changing already-tested code is both risky and costly. If we have unit tests in place, then we can proceed for refactoring confidently.

Reference - Scrapy Contracts

Anmolbansal1 avatar Jan 23 '20 20:01 Anmolbansal1

can i work on this?

carpecodeum avatar Jan 24 '20 22:01 carpecodeum

can i work on this ?

Kunal614 avatar Feb 04 '20 13:02 Kunal614

Yes sure!!

Anmolbansal1 avatar Feb 12 '20 05:02 Anmolbansal1

@Kunal614 any update??

Anmolbansal1 avatar Feb 27 '20 16:02 Anmolbansal1

@Anmolbansal1 my mid-sem was going on , so now i will continue , can you explain what to do.

Kunal614 avatar Feb 27 '20 17:02 Kunal614

@Kunal614 You need set up structure to test scrapy crawlers and ensure that they provide data in proper format and none of the columns are missing. Also integerate them to Travis CI.

Anmolbansal1 avatar Feb 28 '20 05:02 Anmolbansal1