django-dynamic-scraper

django-dynamic-scraper copied to clipboard

Custom HTTP status codes for DDS checker

2

Hi Holger! First of all, thank you for this fantastic piece of software! I'd love for DDS checkers to support custom HTTP status codes. For example, the website that I'm...

WarToaster

How to use the scheduling parameters to only run scraper for only 1 time ?

1

I have tried to read the Documentation and still cannot understand how can I adjust the parameters to only run the scraper for only 1 time. I have a scraper...

vaimdev

If u have configure celery time to search for new jobs < time spider job need to finish, u ll end up with running same spider job multiple times. The...

bezkos

Scrapers Detail Admins Page load too slow

8

If you have many scrapers (around 100) and u try to change one of them then detail page load too slow (around 35 sec). I try to inspect problem with...

bezkos

Support for more than one image attribute

3

Hello, I think it would be interesting to handle more than one IMAGE attribute. I think you could replace your DjangoImagesPipeline by this code : ``` python class DjangoImagesPipeline(ImagesPipeline): def...

Netquin

I was looking for a way to run the spiders in-processes and avoid scrapyd altogether. I looked online and found a way to do it. If this is something you...

scott-coates

Integrate parts of Scrapy CrawlSpider Rule flexibility into DDS

4

With CrawlSpider Rules there exists a more powerful tool in Scrapy (http://doc.scrapy.org/en/0.14/topics/spiders.html) to crawl pages from different urls following a certain pattern than it is actually realized in DDS with...

holgerd77

holgerd77

holgerd77

SystemError: ffi_prep_closure(): bad user_data (it seems that the version of the libffi library seen at runtime is different from the 'ffi.h' file seen at compile-time)

1

I'm getting this error when I try to run the command `scrapy crawl article_spider -a id=1 -a do_action=yes` ``` 2023-04-13 11:14:39 [dds] INFO: 2023-04-13 11:14:39 [dds] INFO: ====================================================================================== 2023-04-13 11:14:39...

natea