apify-sdk-js icon indicating copy to clipboard operation
apify-sdk-js copied to clipboard

Feat: cheerio experimental queue

Open petrpatek opened this issue 1 year ago • 1 comments

This is a POC version with minimal API and minimal request footprint.

Possible improvements:

  • Use magazine instead of the queue to allow syncing with kv-store during the run and save only the rest loaded in memory.
  • compress the requested objects in memory
  • use large-set package to bypass nodeJs internal limit for Set

petrpatek avatar Apr 27 '23 16:04 petrpatek

Let's keep this non-merged for some time.

metalwarrior665 avatar Apr 27 '23 18:04 metalwarrior665