crawler
crawler copied to clipboard
`init_hook` method called during class instantiation
https://github.com/lorien/crawler/blob/master/crawler/base.py#L86
init_hook
method called inside __init__
So for instance if you're doing some work (db calls etc.) inside init_hook
there is no way to change class attributes before calling init hook so the following code will not work.
@cmd.command()
@click.option('--num', type=int, default=1000,
help='Batch size')
def profiles(num):
from crawlers import YourCrawler
bot = YourCrawler(
num_network_threads=10)
bot.number_per_page = num
bot.run()