fleex

fleex copied to clipboard

Current implementation make equal chunks and send for scan. But if one chunk takes more time to complete the job all other machine sits idle. There is an option -d to delete upon completion but this is not helpful if some chunk requires more time.

Instead of making static chunk like lines_count(input_file)/len(fleet), dynamically distributing using a loop by sending few lines for fleet be helpful.

I hope I'm not replying too late I will spend some time to think abount an implemetation for this problem