linkinator icon indicating copy to clipboard operation
linkinator copied to clipboard

linkinator and curl disagree on link status

Open chalin opened this issue 2 years ago • 2 comments

linkinator reports a status of 0 for https://developer.android.com/reference/android/os/Build as shown here:

$ npx linkinator https://developer.android.com/reference/android/os/Build --verbosity DEBUG
🏊‍♂️ crawling https://developer.android.com/reference/android/os/Build
[0] https://developer.android.com/reference/android/os/Build

  [0] https://developer.android.com/reference/android/os/Build
[
  {
    "message": "maximum redirect reached at: https://developer.android.com/reference/android/os/Build",
    "type": "max-redirect",
    "config": {
      "method": "GET",
      "url": "https://developer.android.com/reference/android/os/Build",
      "headers": {
        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36"
      },
      "responseType": "stream",
      "timeout": null
    }
  },
  null
]
ERROR: Detected 1 broken links. Scanned 1 links in 3.258 seconds.

But if I use, say, curl or visit the link manually and look at the network traffic, all I see are 200 status codes. For example:

$ curl -o - -I https://developer.android.com/reference/android/os/Build 
HTTP/2 200 
last-modified: Thu, 10 Feb 2022 17:54:54 GMT
content-type: text/html; charset=utf-8
...

Why the difference?

chalin avatar Aug 16 '22 08:08 chalin

I'm seeing something similar with a microsoft.com link, and a few other sites

$ npx linkinator https://www.microsoft.com/en-us/microsoft-365/excel --verbosity DEBUG
🏊‍♂️ crawling https://www.microsoft.com/en-us/microsoft-365/excel
[0] https://www.microsoft.com/en-us/microsoft-365/excel

  [0] https://www.microsoft.com/en-us/microsoft-365/excel
[
  {
    "message": "network timeout at: https://www.microsoft.com/en-us/microsoft-365/excel",
    "type": "request-timeout",
    "config": {
      "method": "GET",
      "url": "https://www.microsoft.com/en-us/microsoft-365/excel",
      "headers": {
        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36"
      },
      "responseType": "stream",
      "timeout": 20000
    }
  },
  null
]
ERROR: Detected 1 broken links. Scanned 1 links in 20.01 seconds.

vs.

curl -o - -I https://www.microsoft.com/en-us/microsoft-365/excel
HTTP/2 200 
accept-ranges: bytes
content-length: 1020
content-type: text/html
etag: "6082151bd56ea922e1357f5896a90d0a:1425454794"
last-modified: Wed, 04 Mar 2015 07:39:54 GMT
server: AkamaiNetStorage
date: Mon, 16 Jan 2023 17:22:24 GMT

taylorreece avatar Jan 16 '23 17:01 taylorreece

Ditto on this. I'm trying to run linkinator against a localdev site which i can curl without issue, but linkinator gives me a 0 error code.

jerturowetz avatar Mar 22 '24 03:03 jerturowetz