crawlee-python
crawlee-python copied to clipboard
Create a new guide for scaling the crawlers
- We could create a new documentation guide for scaling the crawlers (mainly the features from
_autoscaling
subpackage). - The guide should include the following:
-
ConcurrencySettings
- how users can configure the concurrency of requests. - Add a short explanation of the internal components:
-
Snapshotter
, -
AutoscaledPool
.
-
-
- Inspiration: https://crawlee.dev/docs/guides/scaling-crawlers
- How to name it? I used "request scaling" here, but it is not precise enough.