crawlee-python
crawlee-python copied to clipboard
fix!: Added the always-enqueue paramter to bypass deduplication
Description
- Earlier the requests that were generated from
Request.from_url
with the sameunique_key
generated on the same URLs were considered identical requests but this parameter if set to true allows deduplication of requests by creating a newunique_key
for each request on a consistent URL
Issues
- Closes: #547
Testing
- The changes can be checked by creating requests for the same URL and each time this is performed, it should perform its operations without considering it as an identical request if
always_enqueue
is labelledtrue
Checklist
- [x] CI passed