Update func isDomainAllowed

Open jiguangsdf opened this issue 6 years ago • 3 comments

like scrapy. if c.Visit("http://www.xxxx.com.cn/"), colly.AllowedDomains("xxxx.com.cn") should be work for main domain. so if we request 'www.xxxx.com.cn' and AllowedDomains are 'xxxx.com.cn'. This means that we have access to all subdomains under the top-level domain name

Dec 18 '19 08:12 jiguangsdf

This is a great change, I was just looking for a good solution for this issue. 👍

Jan 03 '20 00:01 maaronking

Any update about this? ...

Jul 14 '21 17:07 edoardottt

I dunno, this is a pretty breaking change. Depending on your use case, you might want this behaviour, or not.

In any case, one can do arbitrary origin check of the URL just before calling Visit, having it built in into Collector is not strictly necessary.

Jul 14 '21 21:07 WGH-