arachne

arachne copied to clipboard

Dynamic-endpoint support

2

It would be nice if I could parse dynamic endpoints(in SPIDER_SETTINGS) like: 'endpoint': 'crawl/'

Strahivan

Hey @kirankoduru , How can I pass any (dynamic) URL to scrape to the spider ?

dalalRohit

I have made a news scraping spider which stores items into an sqlite3 database. I've added the following to the settings.py file `SPIDER_SETTINGS = [ { 'endpoint': 'tech_news', 'location': 'spiders.news_spider',...

zodiac15

Rendered-HTML Views as response

2

After I've hit a request to a certain endpoint, I'll receive a JSON-View. Is there a way for me to render a HTML-file and return it as response?

Strahivan

Fix scrapy settings priority over global settings

3

#### What does this PR do? - Fix the settings defined in **default_settings.py** not being applied. Currently, only settings specified in **SCRAPY_SETTINGS,** which itself is an empty dict are evaluated...

dmkitui

Could you please highlight the importance of a certain project structure in the documentation? It didn't work out for me when I was using the scrapy-given project structure. It threw...

Strahivan

Could you please highlight the fact that the application is not runnable by 'flask run' but 'python .py'

Strahivan

CLOSESPIDER_PAGECOUNT Setting doesn't work for me

1

I added this to my `settings.py` but it doesn't work ```python SPIDER_SETTINGS = [ { 'endpoint': 'dmoz', 'location': 'spiders.dmoz', 'spider': 'DmozSpider', 'scrapy_settings': { 'ITEM_PIPELINES': { 'pipelines.AddTablePipeline': 500 }, 'CLOSESPIDER_PAGECOUNT': 2...

ghostku

Default Settings: Empty SCRAPY_SETTINGS dict.

1

Hi @kirankoduru, I have just started using your module and I may be wrong about this. Here is the situation. In my case I need to specify a pipeline, but...

dmkitui

eventuallyc0nsistent