Pipe input and concurrent futures

[DylanTiger]

• Now it's possible to use pipe for input. Silent flag will be automatically activated.
• Implemented concurrent futures and locks to avoid race conditions
• Implemented batch processing to avoid frequent locks that slow down the script

[rotemreiss]

Thanks for the PR @DylanTiger! I'll review it soon.

When I started working on it a while ago I created some unit tests as well, but I never had the time to complete the job.

I think that we must have unit tests for this change and I wonder if you can handle this part as well or should I try to find the time to handle it? I can share the branch I worked on (with the failing tests).. Any chance that

[DylanTiger]

OK i may try to check...however i already know that there are little discrepancy using this updated script, probably due to the batch logics. I suppose that if the same url is in 2 different batches, it won't be unduplicated so the results may contain few duplicated urls, if you think that this is a problem i may try to solve it applying a second deduplication on the final result.