django-dynamic-scraper icon indicating copy to clipboard operation
django-dynamic-scraper copied to clipboard

Support for more than one image attribute

Open Netquin opened this issue 12 years ago • 3 comments

Hello,

I think it would be interesting to handle more than one IMAGE attribute.

I think you could replace your DjangoImagesPipeline by this code :

class DjangoImagesPipeline(ImagesPipeline):

    def get_media_requests(self, item, info):
        try:
            for img_elem in info.spider.scraper.get_image_elems():
                if img_elem.scraped_obj_attr.name in item and item[img_elem.scraped_obj_attr.name]:
                    yield Request(item[img_elem.scraped_obj_attr.name])
        except (ScraperElem.DoesNotExist, TypeError):
            pass

    def image_key(self, url):
        image_guid = hashlib.sha1(url).hexdigest()
        return '%s.jpg' % (image_guid)

    def thumb_key(self, url, thumb_id):
        image_guid = hashlib.sha1(url).hexdigest()
        return '%s.jpg' % (image_guid)

    def item_completed(self, results, item, info):
        try:
            img_elems = info.spider.scraper.get_image_elems()
        except ScraperElem.DoesNotExist:
            return item
        for i in range(len(results)):
            if results[i][0]:
                print(results[i][1])
                for img_elem in img_elems:
                    if results[i][1]['url'] == item[img_elem.scraped_obj_attr.name]:
                        item[img_elem.scraped_obj_attr.name] = results[i][1]['path']
        return item

Netquin avatar Apr 28 '13 18:04 Netquin

Hi, thanks for your suggestion and sorry for the late answer. I have little time to work on DDS at the moment, but I'll try to have a look at it and integrate in in the next release if I can accept it or I will post here for discussion.

holgerd77 avatar May 24 '13 15:05 holgerd77

Good Job.I need two Image-Fields.

dominicx avatar Aug 11 '14 04:08 dominicx

Will see if I can have a look at it again.

holgerd77 avatar Aug 12 '14 10:08 holgerd77