discover-rewards-notifier icon indicating copy to clipboard operation
discover-rewards-notifier copied to clipboard

Add multithreading to tools/dataProcessor.js

Open nareddyt opened this issue 6 years ago • 0 comments

Currently, the data processor processes deals and cashback rewards one at a time. This takes around 2 minutes to process all data.

It would be significantly faster if this was multithreaded so it can process multiple items at once. Since the majority of time is spent in the Googling hostnames phase (where we have to wait for network responses), it could definitely help.

One concern is that Google might rate limit or even mark us as a bot if we were to increase the speed. Someone will have to try it out and see.

nareddyt avatar Mar 23 '18 15:03 nareddyt