Resolving Slowness

[joshuamcginnis]

I'm using Rack::Cache + Cloudfront over Heroku with the ruby and imagemagick buildpacks. I'm trying to figure out how to speed up my app which has images saved through dragonfly in S3.

It looks like the images are being 304'd, but the initial page load is slow because it appears that, even though the server isn't returning the image in the request, it still must re-convert the image in order to check if it's changed since the last ETag (I'm presuming).

Why am I seeing all of the shell commands being run for every image request? Is it possible to somehow cache this as well?

DRAGONFLY: shell command: 'convert' '/tmp/dragonfly20160611-9-i0yqzv.tif' '-resize' '200x' '/tmp/dragonfly20160611-9-1ygg346.tif'

[markevans]

hi - certainly it shouldn't be run for every request, as it also uses the Cache-Control header, which should tell Cloudfront (and Rack::Cache actually) to cache for however long (specified in header) and not ask about ETag, so something else seems wrong there. After the first time, they should be served directly from Cloudfront without asking your app

[markevans]

also, even if doesn't get cached with Cache-Control for some reason, it doesn't need to re-convert to return the 304. (in short I'm not sure what the problem is but if it's working properly, it should not be hitting your server and running convert after the first time)

[joshuamcginnis]

How does dragonfly keep from having to convert in order to generate the url for each page load?

I have a page that loads 10 images and it takes a good 12 secs for the page request to finish executing and finally load. The actual image loading once the page has rendered is fast because of CloudFront caching, but the execution of the page is slow because of all the calls to convert.

Here's what the page is rendering:

field.image.watermark.thumb('200x').encode('png').url

[markevans]

it doesn't call convert to generate the url. could there be somewhere else on the page it's getting called?

[joshuamcginnis]

I'm probably doing something wrong. Let me see if with adding some context will shed some light.

This isn't the exact implementation of my image_tags as I've abstracted it out, but here's what it boils down to:

<%= link_to image.encode('jpg').url do %>
  <%= image_tag image.thumb('600x').encode('png').url, 
          height: image.thumb('600x').encode('png').height,
          width: image.thumb('600x').encode('png').width %>
<% end %>

This is linking a thumbnail to the full-sized image.

This renders CloudFront link:

http://my_distribution.cloudfront.net/media/long_random_string/file.png?sha=xxxxxxxxxxxx

The rendering of the page triggers multiple converts (for each of the thumb() and height/width calls):

2016-06-12T21:58:09.223322+00:00 app[web.1]: DRAGONFLY: shell command: 'convert' '/tmp/dragonfly20160612-18-ayvgvo.tif' '-resize' '600x' '/tmp/dragonfly20160612-18-1hbrmra.tif'
2016-06-12T21:58:09.452564+00:00 app[web.1]: DRAGONFLY: shell command: 'convert' '/tmp/dragonfly20160612-18-1hbrmra.tif' '/tmp/dragonfly20160612-18-8g0ujh.png'
2016-06-12T21:58:09.659961+00:00 app[web.1]: DRAGONFLY: shell command: 'identify' '-ping' '-format' '%m %w %h' '/tmp/dragonfly20160612-18-8g0ujh.png'
2016-06-12T21:58:09.962238+00:00 app[web.1]: DRAGONFLY: shell command: 'convert' '/tmp/dragonfly20160612-18-44k080.tif' '-resize' '600x' '/tmp/dragonfly20160612-18-q5kfr8.tif'
2016-06-12T21:58:10.241361+00:00 app[web.1]: DRAGONFLY: shell command: 'convert' '/tmp/dragonfly20160612-18-q5kfr8.tif' '/tmp/dragonfly20160612-18-8kkxjw.png'
2016-06-12T21:58:10.489754+00:00 app[web.1]: DRAGONFLY: shell command: 'identify' '-ping' '-format' '%m %w %h' '/tmp/dragonfly20160612-18-8kkxjw.png'
2016-06-12T21:58:10.503566+00:00 app[web.1]: Rendered fields/my_image_field/_show.html.erb (1780.4ms)

Once the page renders, the image loads fine from CloudFront. So it seems something about image_tag is causing the convert to be fired on each page load.

[markevans]

yeh - the height and width calls are doing it - in order to know the width and height it needs to create the thumbnail and identify it. If I was you I'd either just put in a "width:600" attribute, and if I really needed the height one too, save "aspect_ratio" as a magic attribute http://markevans.github.io/dragonfly/models/#magic-attributes and work out height from that (600 / aspect_ratio)

[joshuamcginnis]

Are the calls to .encode and .thumb also triggered on each load? Should those be processed on image save as well?

Is there a definitive list of "Magic" attributes somewhere? The doc doesn't mention aspect_ratio as being one of these magic attributes.

[joshuamcginnis]

I've made some progress on this. I'm now storing a lot of the meta data on my raw uploaded images so that I can avoid the additional .width and .height calls.

  create_table "images", force: :cascade do |t|
    t.string   "attachment_uid"
    t.integer  "attachment_width"
    t.integer  "attachment_height"
    t.integer  "attachment_size"
    t.float    "attachment_aspect_ratio"
    t.boolean  "attachment_portrait"
  end

However, I'm building an app which keeps a repository of thousands of images. I'm running into Heroku R14 - Memory Quota Exceeded in Ruby (MRI) because of the ImageMagick calls and poor garbage collection on Ruby's part.

Some of my images are being shown in up to 6 six different sizes. That's potentially 6*Number of images calls to ImageMagick.

Does anyone have any suggestions about how to improve this? Am I going to need to add a bunch of columns on image table so that I copies / resizes all the various sizes on upload? That seems really ugly.

[joshuamcginnis]

@markevans Any thoughts on the issue outlaid in my last update?