Very choppy scrolling in a ListView with SmartImageView

I have been trying to optimize my ListView for a while now and am failing. When I only set a placeholder image and do not load remotely, the list is very fast.

Here is what I have tried (and still have all this):

• ViewHolder for all views
• ViewHolder stores reference to current URL, so as long as URL didn't change, it won't refresh (this helps if the list refreshes with new data, but the same order of rows)
• Loading thumbnail images (256x256), so minimal resizing or cropping

What else can I try? I assumed the memory cache would be fast enough to make this a non-issue. To get an idea for performance, it is clearly dropping frames and a flick scroll can lock up the list for at least 500-1,000ms.

Thank you for an otherwise awesome library!