node-reddit-scraper icon indicating copy to clipboard operation
node-reddit-scraper copied to clipboard

the app should check if file exists before querying imgur api

Open mikedevita opened this issue 10 years ago • 0 comments

right now the scraper queries reddit for the topics, then for each topic that has an imgur link it queries imgur for the right URL to the image, then it checks if the file exists.

Since imgur has a ratelimit of 12.5k requests a month the app should have some way of determining if a file exists even before querying the imgur API..

maybe by hashing the image and using that as all or part of the filename, or by hashing the URL.

mikedevita avatar Nov 30 '15 19:11 mikedevita