Scrapegraph-ai
Scrapegraph-ai copied to clipboard
feat(parallel-exeuction): parallel asyncio event loop dispatcher
#199
- wrap sequential functions into async equivalents
- seamless event loop handling
- batch size for semaphore not to exhaust too many resources
TODO - still untested
I hope it's fix error: RuntimeError: asyncio.run() cannot be called from a running event loop
in smart_scraper_graph.run()
TODO 🐱
- [x] test it with different batchsizes and system configuration;
- [ ] update docs with benchmarks.
:tada: This PR is included in version 0.11.0-beta.8 :tada:
The release is available on:
-
v0.11.0-beta.8
- GitHub release
Your semantic-release bot :package::rocket:
:tada: This PR is included in version 0.11.0 :tada:
The release is available on:
-
v0.11.0
- GitHub release
Your semantic-release bot :package::rocket: