Not All Thumbnails Getting Generated

[grommett]

Hi,

I'm having an issue where only a few thumbnails are getting generated per build. If I re-run docpad generate or docpad run they all eventually get created. This usually takes a couple of tries to get them all created. Do you have any ideas why this would be happening? This only occurs in my environment (mac os 10.9.1), using imagemagick 6 or graphicsmagick, but works on other computers in my team.

Thanks

[rantecki]

Interesting, I wonder if it has anything to do with the databaseCache setting.

Can you confirm if you have databaseCache: false in docpad.coffee by any chance?

[grommett]

Hi,

I did not initially have databaseCache:false set. However after setting it to false I’m still experiencing the same issue. Since this only happens in my environment, do you think I might be bumping up against a memory issue? We’re converting about 40 images to 4 different sizes when we build. Conversion seems to happen to about 15 per build so it takes roughly 3 docpad run to get them all created.

Thanks

On Feb 23, 2014, at 7:07 PM, Richard A [email protected] wrote:

Interesting, I wonder if it has anything to do with the databaseCache setting.

Can you confirm if you have databaseCache: false in docpad.coffee by any chance?

— Reply to this email directly or view it on GitHub.

[impaler]

I worked around this on a memory limited VPS, by changing the default settings of imagemagick to more conservative settings:

#in my bash script just before I run docpad --generate etc..

export MAGICK_MEMORY_LIMIT=xxx # Use up to *MB of memory before doing mmap
export MAGICK_MAP_LIMIT=xxx    # Use up to *MB mmaps before caching to disk
export MAGICK_AREA_LIMIT=xxx   # Use up to *MB disk space before failure
export MAGICK_FILES_LIMIT=xxx  # Don't open more than *file handles

Maybe this will work for someone else?

@rantecki If this is the root cause, was for me, maybe this issue should be closed.

[tomaspapan]

same problem here, no matter if I use imagemagick 6 or graphicsmagick, it always generate only 79 files, so I have to run it several times to get all thumbnails. It's on Mac (10.9.3), on linux it works just fine. I've tried also what @impaler suggested, but without any luck, the behaviour is the same.

$ identify -list resource

File Area Memory Map Disk Thread Throttle Time

65536 65.536GB 7.6294GiB 1.9073GiB unlimited 1 0 unlimited

[rustycoat]

I'm also experiencing this problem on Mac OS (10.8.5). Seems to have cropped up out of nowhere, but I suspect that it is as @tomaspapan indicated and that I just passed some sort of threshold in terms of the number of images.

I haven't verified whether it's fine running on a linux based systems yet, though I can see that other users (@impaler) had problems in that context. Memory issue or not it's rather hard to track down if I get no error reporting. Is there perhaps a place I should be digging for specific errors (some log hiding away somewhere?) from imagemagick/graphicsmagick on Mac OS?

What concerns me the most is there is no error reporting for the failed generations. I see the images being added to queue and everything appears fine on the console. It was only after deploying that I noticed bad links for some of my older thumbnails.

If I do docpad clean; docpad run I get fairly random results as to which images end up with thumbnails and which images fail to generate.

[rustycoat]

Only indication in console is

info: thumbnail generation complete 
info: Generated 162/181 files in 62.508 seconds

It looks according to the code as if img.quality(...).resize(...).write(dstPath, err) on line 230 is not receiving an error and that's why all of docpad-plugin-thumbnails warnings and errors aren't being triggered.

Unfortunately, I've not much knowledge of how to go about getting this setup properly in a debugger enough to set a breakpoint within the docpad-plugin-thumbnails code...

[priezz]

The problem is caused by file descriptors limit. In previous OS X versions (prior to 10.8.4) is was allowed to set the value to unlimited, and it could be still set to but it does not work. To solve do the following:

1. Append the following lines to /etc/launchd.conf (create if it does not yet exist):

limit maxfiles 10240 10240
limit maxproc 2000 2000

1. Append the following lines to /etc/profile:

ulimit -n 10240
ulimit -u 2000

Bingo!