promise-pool
promise-pool copied to clipboard
supercharge
Is it possible to timeout processing of an individual item?
I'm using promise-pool in my application for controlling the concurrency of a file downloader to avoid triggering the request rate limitations of an upstream API, for which it's working fairly well but I'm troubleshooting a bit of an edge case and need some help from the library.
Problem I'm facing right now is that my pool sometimes develops a "clog" and I need to metaphorically drano the sucker.
Between the pool statistics available to the onTask* callbacks and the index available to the ProcessHandler callback I've been able to add enough logging to those 3 points to determine that I have several items that the pool starts processing but then they never finish and there are no errors reported.
So the pool will end up finishing the last item with 3 Active Tasks, N-3 Finished Tasks, and then hang forever. In the interim while I don't know what's going on, I'd love to be able to timeout processing of an individual item as a last resort or as a simple way to retry entire items via the library without having to blow out the size and responsibility of my ProcessHandler code to write my timeout and retry manually in there.
@pandrews-hdai Hey, at this point it's not possible to timeout an item. Also the pool won't timeout.
I appreciate your help if you want to submit a pull request adding this feature.
I'm currently on vacation and will be for the next week. Maybe you can help yourself in the meantime. Otherwise I can look at it in two weeks.
@marcuspoehls Good to know. I may have some time to tinker with it. I assume you'd prefer to avoid introducing additional dependencies so if I do put time into a patch it should be baked in and not provided by a 3rd-party module?
@pandrews-hdai I prefer things to be baked in. And baking in if it makes sense. For example, my guess is the timeout could be baked in. For a retry (if we add that logic as well), it makes sense to add a library. Because retry is solved already in other 3rd-party packages that are maintained and have good code 😃
@pandrews-hdai Have you been able to tinker with the timeout implementation? Can you please share an update?
@marcuspoehls Unfortunately no, I'm US based so I elected to take the holiday and just haven't had the spare time to prioritize anything around this issue afterwards. Though I'm blocked waiting for a few bits of infrastructure so I'll probably take some day-job time today.
@marcuspoehls I've built a working timeout in my own code and have been trying to patch it into your code for the pool as well but the test isn't working correctly yet despite working during run-time testing my application with the local build of the promise-pool with the timeout feature.
I think this will be a lifesaver and I am having big issue with large data choking my server. @pandrews-hdai if I understand correctly, this timeout processing of an individual item will put some sort of delay between an item to finish before the next?
@marcuspoehls can you please merge?
@charlesmudy I’ll have a look at this in the afternoon.
This timeout feature ensures that the Promise for a task (task = running the process function for an item) is limited to the given timeout. The processing must finish within the time of the configured timeout. Otherwise the pool errors out for the item.
Wonderful, this just happened to me and I was dreading having to deal with this internally. So happy to see the PR 🎉
Forked the lib and brought this feature into our app and and can confirm that it fixes a major frequent issue we were facing when working with a very large dataset 👍 Super large collections no longer stall out. Looking forward to the latest release of the lib.
I’ve published version 2.4.0 to NPM which contains the .withTaskTimeout method 😃
@pandrews-hdai thank you very much for your help on this!