django-imagekit
django-imagekit copied to clipboard
matthewwithanm
Performance.
Hello,
I have successfully installed and run imagekit adding an ImageSpecField Now my rest service is timing out. I have a django server on heroku and s3 storage for my image. Is there a set up I'm missing here?
To be clear, all other rest services that I'm calling that do not return images are working fine.
As you mentioned in previous issue you are starting to use imagekit and nothing is generated already in your S3 storage. S3 is not like local filesystem and if your images are big an you need to create a lot of thumbnails per request it can timeout.
How many images need to be generated per request? Is this a public production API or this happens on your development heroku instance (how many paralel requests are comming to your API)?
Hello,
Thanks for your response. From your previous response and reading the doc I understood that it will take awhile to create the new thumbnails. So I wrote a script to query my api and reach all my images. Which successfully created all 500 thumbnails (after lots of timeout indeed :))
Now All thumbnails have been created but my api (using djangorest) is still slow.
Model looks like this: itempic = models.ImageField(upload_to=upload_to, blank=True, null=True)
itempic_small = ImageSpecField(source='itempic', processors=[ResizeToFit(1000, 1000)], format='JPEG', options={'quality': 60})
In my rest serializer I call: itempic.url
On Sun, Jul 3, 2016 at 11:38 PM, Venelin Stoykov [email protected] wrote:
As you mentioned in previous issue you are starting to use imagekit and nothing is generated already in your S3 storage. S3 is not like local filesystem and if your images are big an you need to create a lot of thumbnails per request it can timeout.
How many images need to be generated per request? Is this a public production API or this happens on your development heroku instance (how many paralel requests are comming to your API)?
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/matthewwithanm/django-imagekit/issues/375#issuecomment-230215857, or mute the thread https://github.com/notifications/unsubscribe/AGj12XjDz5kQyOxYAzfw5vSbo_u-uMRkks5qSKpLgaJpZM4JDm0z .
itempic.url or itempic_small.url?
If you are using itempic.url only the slowness is entirely in communication with S3.
If you misstyped it in the comment and actually you are using itempic_small.url than can I ask you to add similar API which is different from the original only by replacing itempic_small.url with itempic.url and see the performance impact. If with itempic.url the performance is slow then the problem is not in django-imagekit.
It's obvious that there will be some performance overhead with using django-imagekit but when images are generated it must be negligible.
One more thing. You are using latest version and you use shared cache between your dynos right?
Hello,
This was a typo yes . This is why I'm reaching out b/c the test I did was to use itempic_small.url and back to itempic.url and the performance are back to normal. Otherwise I wouldn't have bugged you. my version is 3.3 and I have a hobby heroku box. I must not be the only one with 500 pics and a heroku box, is there any setting I'm missing on imagekit?
Hello,
It's good that you already made this test. One last question. You are trying to get thumbnails of 500 images at once, or all of your images are around 500 and you have some sort of paging?
In normal situations I'm not sure if django hit's the filesystem to generate the urls, but with imagekit if it is not in the cache it need to hit the storage to check if the generated file is there. 500 hits to S3 is a lot per single request.
As I said about caching how you configured your cache?
One more thing. Can you try the code from #380 to see if there are any difference?
Actually I'm not using S3 and I haven't any observations about it. Some contribution to documentation from someone who has experience or even PR with fixes if possible will be welcomed.
+1 Similar setup, it takes about 20 seconds (of which 18s "waiting" when analyzing network) to upload a 1.7MB picture. I do have 6 ImageSpecFields, mainly resizing. I tried to do it through Celery but didn't see much improvements.
Hello guys. I'm introducing new default behavior for ImageKit caching which I hope will benefit your situation (#403).
Currently I'm not sure if people reading the documentation (http://django-imagekit.readthedocs.io/en/stable/caching.html#optimizing) are understanding correctly what they need to do in order ImageKit to use the cache and use it efficiently.
In #402 we discussed probably similar problem. You can see my comment there which gives example config of caches https://github.com/matthewwithanm/django-imagekit/issues/402#issuecomment-278580446. I hope that this config can reduce calls to S3 and overall response time, after images are once created.
If you can test this config and if it helps then changes in #403 will be sane default:
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
'LOCATION': '127.0.0.1:11211',
},
'imagkit': {
'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
'LOCATION': '127.0.0.1:11211',
'TIMEOUT': 7 * 24 * 60 * 60, # One week
},
}
IMAGEKIT_CACHE_BACKEND = 'imagkit'
I'm having a similar issue with ImageKit 4.0.2.
image = ImageField
image_thumb = ImageSpecField
image.url works quickly while image_thumb.url is incredibly slow. I'm using django-storages 1.6.6 and s3. If both fields query s3 for the url, why would image_thumbnail.url be significantly slower?
I think this problem should be addressed regardless of the cache. You can imagine having >10,000 images in which case it might not be a good idea to store all the image keys in something like memcache (I've got multiple ImageSpecs per image, and use the cache for many other things).
Sorry this answers my question,
You might be wondering why we didn't need an "upload_to" argument for our ImageSpecField. The reason is that ProcessedImageFields really are just like ImageFields—they save the file path in the database and you need to run syncdb (or create a migration) when you add one to your model.
ImageSpecFields, on the other hand, are virtual—they add no fields to your database and don't require a database. This is handy for a lot of reasons, but it means that the path to the image file needs to be programmatically constructed based on the source image and the spec.
Perhaps I don't need the virtual images, or I could get them asynchronously :/
You can try to use celery to create them async. Then ImageKit does not need to check if the file is in the storage but will return the URL directly and hopes that the thumbnail is there.
There can be a problem though (look at #437). I'm working on merging this soon and releasing new version in order not to be a problem. Until then if you have the problem described in #437 you can use that branch to check if it will work for you.