fastclass icon indicating copy to clipboard operation
fastclass copied to clipboard

Published 20 hours ago • verified publisher icon cwerner

Scraping Pinterest?

Open venuv opened this issue 5 years ago • 11 comments

First of all, nicely done (good docs, easy install, works like a charm). Would find pinterest scraping quite useful ..

venuv avatar Oct 18 '19 02:10 venuv

Hi.

Thanks for using the tool. I have not had the use case myself, but if there are easy options to include this - why not? However, the tool piggy-backs on icrawler which only deals with Google, Bing and Baidu. Would you know of the required tools for this?

cwerner avatar Oct 18 '19 09:10 cwerner

I could manually search for the Pinterest board of relevance to the keyword(s) I'm searching (say on 'recliners') and put the url in the excel file. Then fastclass could crawl that using a capability such as https://github.com/xjdeng/pinterest-image-scraper

venuv avatar Oct 18 '19 14:10 venuv

Interesting.

I quickly peeked into the mentioned package and unfortunately this requires selenium which is a pretty hefty burden for such a small package. I will think about it. Would be happy to look at a PR though if you want to give it a shot...

cwerner avatar Oct 18 '19 14:10 cwerner

How can I use the tool to image process a of my own images in a folder? I can't seem to get it to work (I'm new)

oezeadi avatar Mar 30 '20 21:03 oezeadi

Hi @oezeadi ,

Not quite sure what you are trying to do? If you want to classify a bunch of images you use the fc_clean command...

cwerner avatar Mar 30 '20 22:03 cwerner

I have a folder of my own images and I'd like to process that folder as I would using the fcd after it crawls..

oezeadi avatar Mar 30 '20 22:03 oezeadi

Let me get back to you tomorrow... I’ll have a look at it 👨‍💻

cwerner avatar Mar 30 '20 23:03 cwerner

ok thanks, can't wait!

oezeadi avatar Mar 31 '20 01:03 oezeadi

Ok @oezeadi

I have some unfinished changes to fcc (there're also tk ui bugs I want to fix), but I just gave the current GitHub status a spin and I get the result that I expect...

So if you want to assign images to certain classes you start the command with

fcc folder/where/your/files/are

Then you use the numbers 1-9 to assign a class number to the image. You will automatically move to the next after press... If you need to change the assignment you can use the left/ right arrow keys. Once done you save the assignment by pressing the X key.
You should get a report file in the folder which has the filename and the assigned class in there.

Is this what you are seeing, too?

cwerner avatar Mar 31 '20 16:03 cwerner

Hey, so I guess my first question should have been: does your program actually do other types of image processing (resizing, checking for right number of channels, remove duplicates, etc)? That is what I assumed fcd was doing after it crawled for images.

If yes to above, then does the new fcc you updated do these checks too? I don't know how to tell if a batch of images have been properly "checked/process"...

oezeadi avatar Apr 05 '20 20:04 oezeadi

Ah ok.

Right, this would make sense. I anticipated fcc to weed through the download and basically mark bad or misclassified ones (so run after fcd). I’ll have a look how to share these checks between the scripts. A bit busy next week but hope to update soon. However, if you have specific needs/ ideas I would also appreciate a PR 😉

C

cwerner avatar Apr 05 '20 22:04 cwerner