image-match icon indicating copy to clipboard operation
image-match copied to clipboard

Fast Insert

Open Kencho5 opened this issue 2 years ago • 2 comments

as you mentioned in the readme, it states that it can easily scale up to billions of images, but i'm having a hard time inserting a lot of images in elasticsearch. i've tried to do it with multiprocessing, threading and even async too, but given that i have around 160 million images it's still slow. any suggestions on how i can insert those images a bit faster?

Kencho5 avatar Nov 30 '21 14:11 Kencho5

did you come up with a solution?

trashmit avatar Apr 12 '22 09:04 trashmit

yes. made bulk insert script, but don't have it anymore. you have to split functions and use multiprocessing

Kencho5 avatar Apr 13 '22 09:04 Kencho5