Error "Callback was already called." when setting httpHeaders in mlc_config.json

[ndsvw]

I'm trying to check the links in this file (state 30.10.22): https://raw.githubusercontent.com/luong-komorebi/Awesome-Linux-Software/master/README_pt-BR.md

It works fine as long as I don't set HTTP headers in mlc_config.json.

Command: markdown-link-check ./README_pt-BR.md --config ./mlc_config.json (latest version - 3.10.3)

Result:

FILE: ./README_pt-BR.md /usr/local/lib/node_modules/markdown-link-check/node_modules/async/dist/async.js:324 if (fn === null) throw new Error("Callback was already called."); ^

Error: Callback was already called. at /usr/local/lib/node_modules/markdown-link-check/node_modules/async/dist/async.js:324:36 at /usr/local/lib/node_modules/markdown-link-check/node_modules/async/dist/async.js:248:17 at /usr/local/lib/node_modules/markdown-link-check/index.js:129:13 at /usr/local/lib/node_modules/markdown-link-check/node_modules/link-check/lib/proto/http.js:117:21 at done (/usr/local/lib/node_modules/markdown-link-check/node_modules/needle/lib/needle.js:548:14) at PassThrough. (/usr/local/lib/node_modules/markdown-link-check/node_modules/needle/lib/needle.js:811:9) at PassThrough.emit (node:events:525:35) at endReadableNT (node:internal/streams/readable:1359:12) at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

Node.js v18.11.0

It works fine with this mlc_config.json:

{
  "timeout": "20s",
  "retryOn429": true,
  "retryCount": 5,
  "config-file": "mlc_config.json",
  "fallbackRetryDelay": "60s",
  "aliveStatusCodes": [200, 206, 403]
}

But it crashes with this mlc_config.json that has httpHeaders specified:

{
  "timeout": "20s",
  "retryOn429": true,
  "retryCount": 5,
  "config-file": "mlc_config.json",
  "fallbackRetryDelay": "60s",
  "aliveStatusCodes": [200, 206, 403],
  "httpHeaders": [
    {
      "urls": ["http", ".", "/"],
      "headers": {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36",
        "Upgrade-Insecure-Requests": "1",
        "Connection": "keep-alive",
        "Accept-Language": "en-US,en;q=0.5",
        "Accept-Encoding": "gzip, deflate, br",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"
      }
    }
  ]
}

[github-actions[bot]]

This issue has been marked as stale because it has been open 60 days with no activity. It will be closed in 30 days unless the stale label is removed or someone adds a comment.

[ndsvw]

comment

[smainil]

@ndsvw can you check if you can reproduce your issue with the latest version

[github-actions[bot]]

This issue has been marked as stale because it has been open 60 days with no activity. It will be closed in 30 days unless the stale label is removed or someone adds a comment.

[ndsvw]

@ndsvw can you check if you can reproduce your issue with the latest version

@smainil, ok.

I'm a bit confused...

I spawned a node container, because I don't have node anymore:

docker run -it --rm node:latest /bin/bash

Then, I installed it:

npm install -g markdown-link-check

Then I tried checking the links without any mlc config:

FILE: https://raw.githubusercontent.com/luong-komorebi/Awesome-Linux-Software/master/README_pt-BR.md
  No hyperlinks found!

  0 links checked.

It can't find the links anymore. There are plenty in that file...

When using https://github.com/tcort/markdown-link-check/blob/master/README.md, it finds the links.

Version is 3.13.7. Do you want me to open a new issue for this?

[BaseMax]

Hi Alex (@ndsvw), if you are available or able to test it, please let us know. Thanks.

CC https://github.com/tcort/markdown-link-check/pull/540

[BaseMax]

Close - from https://github.com/tcort/markdown-link-check/pull/540

[ndsvw]

Hi Alex (ndsvw), if you are available or able to test it, please let us know. Thanks.

Hi @BaseMax, I'm available.

I tried out the repro steps myself, but unfortunately, I still can't run them, because something else seems to be broken.

Please run these commands:

docker run -it --rm node:latest /bin/bash
npm install -g markdown-link-check
markdown-link-check "https://raw.githubusercontent.com/luong-komorebi/Awesome-Linux-Software/master/README_pt-BR.md"

The problem: markdown-link-check does not even find the links in that file anymore. This is my output of the last command:

root@15c0d1ced677:/# markdown-link-check README_pt-BR.md 

FILE: README_pt-BR.md
  No hyperlinks found!

  0 links checked.

The version used this time is 3.14.2.