colly icon indicating copy to clipboard operation
colly copied to clipboard

Endlessly redirected links fail in colly

Open saswatamcode opened this issue 3 years ago • 0 comments

We're trying to build some link checking functionality using colly. But colly on encountering links like "https://www.envoyproxy.io" gives a "301 moved permanently" error even though it is perfectly viewable in a browser. Curl output for this is given below,

curl -i https://www.envoyproxy.io/
HTTP/2 301 
cache-control: public, max-age=0, must-revalidate
content-length: 42
content-security-policy: frame-ancestors 'self';
content-type: text/plain
date: Tue, 18 May 2021 04:32:10 GMT
x-nf-request-id: fd4710cb-689c-4c75-b8f6-808272b5732b
location: https://www.envoyproxy.io/
server: Netlify
age: 25713

Redirecting to https://www.envoyproxy.io/

Is there any particular fix for this? Related issue

saswatamcode avatar May 19 '21 04:05 saswatamcode