How to set up the robots protocol in Crawlee for Python?

Open CodeDan-zld opened this issue 9 months ago • 1 comments

When using the Python version of Crawlee, I didn't see any options to configure the robots protocol, but I found relevant information in the Node.js version's documentation. If such a configuration exists, please help me.

Mar 07 '25 07:03 CodeDan-zld

Hello, and thank you for your interest in Crawlee! This feature is not yet implemented, but it is on the roadmap.

Mar 07 '25 11:03 janbuchar

@Mantisus This is resolved, am I correct?

Jul 08 '25 14:07 vdusek

@Mantisus This is resolved, am I correct?

Yep

Jul 08 '25 14:07 Mantisus