Memory Leak with easy_thumbnails.files.generate_all_aliases

[epicserve]

I wrote the following script to test a memory leak I was experiencing when running generate_all_aliases.

from mysite.myapp.models import Photo
from easy_thumbnails.files import generate_all_aliases
import time


def time_function(func, *args, **kwargs):
    st = time.time()
    func(*args, **kwargs)
    print('It took %.2f seconds to generate all thumbnails.' % (time.time() - st))


pic = Photo.objects.get(pk=3)
time_function(generate_all_aliases, *[pic.thumbnail], **{'include_global': True})

The script takes 75.72 seconds to generate all the thumbnails and before quiting the python process grows to 1.2 GB in size.

My thumbnail aliases are set to the following ...

THUMBNAIL_ALIASES = {
    '': {
        'admin_thumbnail': {'size': (120, 80), 'crop': 'scale', 'quality': 80},
        'admin_thumbnail_small': {'size': (50, 50), 'crop': 'scale', 'quality': 80},
        'admin_thumbnail_cropped': {'size': (120, 120), 'crop': 'crop', 'quality': 80},
        'profile_photo_thumbnail': {'size': (170, 170), 'crop': 'crop', 'quality': 80},
        'event_thumbnail': {'size': (160, 120), 'crop': 'crop', 'quality': 90},
        'article_photo_thumbnail': {'size': (210, 280), 'quality': 80},
        'photoset_large': {'size': (555, 370), 'quality': 80},
        'photoset_small': {'size': (58, 40), 'crop': 'crop', 'quality': 80},
        'large_photo': {'size': (960, 720), 'quality': 80},
    },
}

The image file was a PNG and was only 393 KB in size so it's not even that big.

I'm using easy-thumbnails==1.2, django-storages==1.1.4, boto==2.8.0 with the following relevant settings ...

# AWS SETTINGS
from S3 import CallingFormat
STATIC_URL = 'http://my-awesome-bucket/static/'
MEDIA_URL = 'http://my-awesome-bucket/media/'
DEFAULT_FILE_STORAGE = 'utils.s3utils.MediaRootS3BotoStorage'
STATICFILES_STORAGE = 'utils.s3utils.StaticRootS3BotoStorage'
THUMBNAIL_DEFAULT_STORAGE = DEFAULT_FILE_STORAGE
AWS_S3_SECURE_URLS = False
AWS_QUERYSTRING_AUTH = False
AWS_ACCESS_KEY_ID = 'xxxxxxxxxxxx'
AWS_SECRET_ACCESS_KEY = 'xxxxxxxxxxxx'
AWS_STORAGE_BUCKET_NAME = 'my-awesome-bucket'
AWS_PRELOAD_METADATA = True
AWS_CALLING_FORMAT = CallingFormat.SUBDOMAIN
AWS_IS_GZIPPED = True
AWS_HEADERS = {
    'Expires': 'Thu, 19 Apr 2040 20:00:00 GMT',
    'Cache-Control': 'max-age=86400',
}

[SmileyChris]

Is this with DEBUG = False?

[epicserve]

@SmileyChris,

I've tried with both DEBUG = False and with DEBUG = True. No change.

[epicserve]

@SmileyChris

I created an example project that shows what I'm finding. The README tells you how to setup the example app and also has a section that describes the results I had.

It would be super cool if people wanted to try out my example app in order to help me debug whats going on and fix this issue.

[jorrete]

Any news about this issue? It's confirmed? I think I've seen the same bug.

[aparakian]

I've encountered the same issue on my project. It's pretty much confirmed as you can see here: I've tried to find the origin of it but all the open files seem closed, it's very confusing. In any cases, be very careful when migrating.

[aparakian]

If I print a traceback of this memory with tracemalloc here is what I get actually:

File "/site-packages/PIL/Image.py", line 3327
    info.load(self.fp)
  File "/site-packages/PIL/Image.py", line 3363
    ifd = self._get_ifd_dict(self[0x8769])
  File "/site-packages/PIL/JpegImagePlugin.py", line 504
    return self.getexif()._get_merged_dict()
  File "/site-packages/PIL/JpegImagePlugin.py", line 474
    return _getexif(self)
  File "/site-packages/easy_thumbnails/utils.py", line 109
    exif = im._getexif()
  File "/site-packages/easy_thumbnails/source_generators.py", line 34
    image = utils.exif_orientation(image)
  File "/site-packages/easy_thumbnails/engine.py", line 110
    image = generator(source, **processor_options)
  File "/site-packages/easy_thumbnails/files.py", line 383
    fail_silently=silent_template_exception)
  File "/site-packages/easy_thumbnails/files.py", line 515
    silent_template_exception=silent_template_exception)
  File "/site-packages/easy_thumbnails/files.py", line 91
    thumbnailer.get_thumbnail(options)
  File "generate_pictures_from_url.py", line 50
    generate_all_aliases(item.picture, include_global=False)