openfoodfacts
Multithreading support for the Python server
Hey, i'm currently throwing tons of queries at my local off database and it seems like just one of my cpu's is being used, is there an easy way to add multithreading or should i just start multipile instances and throw qeries in different directions? :D would that even work?
ok, tested with 8 threads asking one instance, not getting any performance boost, maybe 8 threads asking 8 instances of APIRestPython...
What kind of queries are you making and why are you making tons of them? It might be easier to craft one mongodb query that returns all the products and data you are interested in. That should be the job of MongoDB. The API is more geared to return real time results for live user queries (e.g. scans etc.).
i have a rough database of a few thousand names and brands for my masters project, and i want to throw them at off to get the gtins and all the other sweet info :D i think it might be hard to do those queries with multiple rows at the same time..
Another approach could be to just use the full CSV dump, and iterate line by line over it, keep the data you need for each brand etc.
oh, i haven't thought about that, maybe that would work :) will investigate on that later, on the other hand, running 8 instances of this python api with different ports works :D, but a quick change in the python code doesn't work as one library only works in the main thread :D