discover-rewards-notifier
discover-rewards-notifier copied to clipboard
Add multithreading to tools/dataProcessor.js
Currently, the data processor processes deals and cashback rewards one at a time. This takes around 2 minutes to process all data.
It would be significantly faster if this was multithreaded so it can process multiple items at once. Since the majority of time is spent in the Googling hostnames phase (where we have to wait for network responses), it could definitely help.
One concern is that Google might rate limit or even mark us as a bot if we were to increase the speed. Someone will have to try it out and see.