crawlpy icon indicating copy to clipboard operation
crawlpy copied to clipboard

Published 20 hours ago • verified publisher icon cytopia

__ROADMAP__

Open cytopia opened this issue 8 years ago • 1 comments

Roadmap

v0.4

  • [ ] First parse JS, then crawl (possibly via phantomJS)

v0.3

  • [ ] Make self-contained spider

v0.2

Release v0.2

  • [X] ~~Be able to specify http status codes allowed for crawling (not just only 200)~~
  • [X] ~~overwrite internal DEPTH_LIMIT~~
  • [x] ~~Ignore list of url patterns~~

v0.1

Release v0.1

  • [X] ~~Make it work~~

cytopia avatar Aug 04 '16 13:08 cytopia

Hi! I can help.

DjaPy avatar Nov 30 '18 04:11 DjaPy