Scrapegraph-ai icon indicating copy to clipboard operation
Scrapegraph-ai copied to clipboard

feat(parallel-exeuction): parallel asyncio event loop dispatcher

Open DiTo97 opened this issue 9 months ago • 1 comments

#199

  • wrap sequential functions into async equivalents
  • seamless event loop handling
  • batch size for semaphore not to exhaust too many resources

TODO - still untested

DiTo97 avatar May 10 '24 22:05 DiTo97

I hope it's fix error: RuntimeError: asyncio.run() cannot be called from a running event loop in smart_scraper_graph.run()

Kingki19 avatar May 13 '24 04:05 Kingki19

TODO 🐱

  • [x] test it with different batchsizes and system configuration;
  • [ ] update docs with benchmarks.

PeriniM avatar May 13 '24 16:05 PeriniM

:tada: This PR is included in version 0.11.0-beta.8 :tada:

The release is available on:

Your semantic-release bot :package::rocket:

github-actions[bot] avatar May 13 '24 21:05 github-actions[bot]

:tada: This PR is included in version 0.11.0 :tada:

The release is available on:

Your semantic-release bot :package::rocket:

github-actions[bot] avatar May 14 '24 15:05 github-actions[bot]