grunt-glotpress icon indicating copy to clipboard operation
grunt-glotpress copied to clipboard

Add batchSize option to limit concurrent downloads

Open herregroen opened this issue 5 years ago • 2 comments

Recently we ran into an issue where our GlotPress installation would suffer whenever this task was ran as it attempts to download all files simultaneously, causing a large amount of executions on what is, apparently, a rather expensive operation.

To fix this I've updated the code to support a new batchSize option which limits the number of concurrent downloads accordingly. The default of -1 retains the current behaviour of downloading all files concurrently.

In order to keep code readable I've used async and await to provide this feature along with several other modern JS features such as arrow functions, const and let. Not sure what Node version this package is targeting so it may not be desirable to merge this feature as-is. Did want to share it in any case should anyone else find it useful.

herregroen avatar Apr 08 '19 14:04 herregroen

Thank you for the code. Currently traveling so it will take me a couple of days to look into the code.

markoheijnen avatar Apr 10 '19 08:04 markoheijnen

@markoheijnen have you managed to take a look yet? :)

remkus avatar Apr 18 '19 13:04 remkus