django-dynamic-scraper
django-dynamic-scraper copied to clipboard
Support for more than one image attribute
Hello,
I think it would be interesting to handle more than one IMAGE attribute.
I think you could replace your DjangoImagesPipeline by this code :
class DjangoImagesPipeline(ImagesPipeline):
def get_media_requests(self, item, info):
try:
for img_elem in info.spider.scraper.get_image_elems():
if img_elem.scraped_obj_attr.name in item and item[img_elem.scraped_obj_attr.name]:
yield Request(item[img_elem.scraped_obj_attr.name])
except (ScraperElem.DoesNotExist, TypeError):
pass
def image_key(self, url):
image_guid = hashlib.sha1(url).hexdigest()
return '%s.jpg' % (image_guid)
def thumb_key(self, url, thumb_id):
image_guid = hashlib.sha1(url).hexdigest()
return '%s.jpg' % (image_guid)
def item_completed(self, results, item, info):
try:
img_elems = info.spider.scraper.get_image_elems()
except ScraperElem.DoesNotExist:
return item
for i in range(len(results)):
if results[i][0]:
print(results[i][1])
for img_elem in img_elems:
if results[i][1]['url'] == item[img_elem.scraped_obj_attr.name]:
item[img_elem.scraped_obj_attr.name] = results[i][1]['path']
return item
Hi, thanks for your suggestion and sorry for the late answer. I have little time to work on DDS at the moment, but I'll try to have a look at it and integrate in in the next release if I can accept it or I will post here for discussion.
Good Job.I need two Image-Fields.
Will see if I can have a look at it again.