Pagination with "Response contains next URL" erroring with Missing Authentication

[pcxnet]

Describe the bug Without pagination turned on, it works, but turning it on resuluts in error saying Missing Authentication, but console shows the auth header being sent.

To Reproduce { "meta": { "instanceId": "09ae26421a658d6d4bd137affcc05c36ff6c0d940d6ccd3c5f074ff305860878" }, "nodes": [ { "parameters": { "url": "=https://api.ento.com/au/api/shifts?filter[start_date]=2023-11-15&filter[end_date]=2023-11-15&page[size]=25", "authentication": "genericCredentialType", "genericAuthType": "httpHeaderAuth", "sendHeaders": true, "headerParameters": { "parameters": [ { "name": "Content-Type", "value": "application/vnd.api+json" } ] }, "options": { "response": { "response": { "responseFormat": "json" } }, "pagination": { "pagination": { "paginationMode": "responseContainsNextURL", "nextURL": "=https://api.ento.com/au{{ $response.json.links.next}}", "limitPagesFetched": true, "maxRequests": 2 } } } }, "id": "0747f03b-d9b7-45b8-9b63-cfcca6a1a44b", "name": "getShifts", "type": "n8n-nodes-base.httpRequest", "typeVersion": 4.1, "position": [ 660, 220 ], "credentials": { "httpHeaderAuth": { "id": "74bSgPs38yQaQ3gh", "name": "Ento" } } } ], "connections": {} } I limited pages to 2 just to test.

Expected behavior Cycles thru pages properly.

Environment (please complete the following information):

• OS: Docker
• n8n Version 1.16.0

[Joffcom]

Hey @pcxnet,

Thanks for reporting this, it looks like this issue may have been fixed in 1.17.0 can you try updating your n8n instance and see if that helps.

[pcxnet]

Hi @Joffcom that didn't fix it, but i wonder if it has anything to do with the way the data is returned. In the older version of the Http request, the data showed up properly, but in the new version of http request, the data looks like this

Unless I turn on Response Format - JSON

The dev console doesn't show multiple requests so I can't see what its attempting to request, it only shows the initial one.

[Joffcom]

Hey @pcxnet,

Is it still showing that the authentication is missing?

Which version did you upgrade to in the end as well I can give it a test this morning when I start.

[pcxnet]

Hi @Joffcom I tried 1.17.0 and now even 1.18.0 and both are doing the same thing .

[Joffcom]

Hey @pcxnet,

I just gave it a quick test on 1.17.0 and pointed the http request to webhook.site to chcek if the headers are being sent and it looks like they are there. For my test I used the workflow below which sends the request and I can see the 2 http requests with the test header credential I was sending. I would be surprised if the response content was making a difference to this, Are you able to do a similar test to what I have done to see if the headers are being sent as well that would help us work out what could be going on.

{
  "meta": {
    "instanceId": "8c8c5237b8e37b006a7adce87f4369350c58e41f3ca9de16196d3197f69eabcd"
  },
  "nodes": [
    {
      "parameters": {
        "url": "https://webhook.site/28c6a284-05de-418e-a684-f261a81ce4e9/au/api/shifts?filter[start_date]=2023-11-15&filter[end_date]=2023-11-15&page[size]=25",
        "authentication": "genericCredentialType",
        "genericAuthType": "httpHeaderAuth",
        "sendHeaders": true,
        "headerParameters": {
          "parameters": [
            {
              "name": "Content-Type",
              "value": "application/vnd.api+json"
            }
          ]
        },
        "options": {
          "pagination": {
            "pagination": {
              "paginationMode": "responseContainsNextURL",
              "nextURL": "https://webhook.site/28c6a284-05de-418e-a684-f261a81ce4e9/au/a",
              "limitPagesFetched": true,
              "maxRequests": 2
            }
          }
        }
      },
      "id": "7d1a66ae-76fa-46be-b3cd-e00a33779374",
      "name": "getShifts",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.1,
      "position": [
        720,
        420
      ],
      "credentials": {
        "httpHeaderAuth": {
          "id": "248",
          "name": "TEST"
        }
      }
    },
    {
      "parameters": {},
      "id": "71ec7c31-28a9-4065-9eb1-ca46707ba284",
      "name": "When clicking \"Execute Workflow\"",
      "type": "n8n-nodes-base.manualTrigger",
      "typeVersion": 1,
      "position": [
        460,
        420
      ]
    }
  ],
  "connections": {
    "When clicking \"Execute Workflow\"": {
      "main": [
        [
          {
            "node": "getShifts",
            "type": "main",
            "index": 0
          }
        ]
      ]
    }
  }
}

[pcxnet]

@Joffcom yes this webhook.site test works fine. Can i give you access to my server or something via private message to look at it? I'm not sure exactly what is causing the problem. Im wondering if its because the data isn't readable unless I tick the JSON response box, and its not being parsed before it reads the next page URL or something so its not finding URL or something? I'm not sure. I can screen share with you or whatever you need if you're available now.

[Joffcom]

Hey @pcxnet,

Ah it could be the $response.json.links.next actually, Can you remove the Content-Type header you are setting and see if that changes anything?

[pcxnet]

I get this still

If i point the next URL to itself get this message, which makes me think its not getting the $response properly. Without the header i get this message (which appens even without pagination) and with the content-type header back in, i get this

[Joffcom]

Hey @pcxnet,

It could be that it isn't get the next url properly, Do you have a link to the API docs for the service you are using?

[pcxnet]

Thanks @Joffcom It is very badly written and has very little detail. https://docs.api.ento.com/

but yeh if i dont choose "JSON" as the response type in the new version of http request , it shows all the data together in one huge lump, but on the V1 of http request, it works without doing that.

[Joffcom]

I have not seen any other API return data in that way but it could be the application/vnd.api+json option which is a bit special. Does it improve if you set the content type header to just application/json? We may need to add more support for the format being used but in theory you could try parsing it with {{ JSON.parse($response.body.data).links.next }} and see if that works.

[pcxnet]

With these settings { "meta": { "instanceId": "7286cb7af5e69b909e9502fd59ea92cbd7d4a06c840dc2904f09626675230c35" }, "nodes": [ { "parameters": { "url": "=https://api.ento.com/au/api/shifts?filter[start_date]=2023-11-15&filter[end_date]=2023-11-28&filter[area_id]=76715&page[size]=100", "authentication": "genericCredentialType", "genericAuthType": "httpHeaderAuth", "sendHeaders": true, "headerParameters": { "parameters": [ { "name": "content-type", "value": "application/vnd.api+json" } ] }, "options": { "response": { "response": { "responseFormat": "json" } } } }, "id": "207e34ce-8469-4db6-bc0f-ccab43d11577", "name": "getShifts1", "type": "n8n-nodes-base.httpRequest", "typeVersion": 4.1, "position": [ 640, -20 ], "credentials": { "httpHeaderAuth": { "id": "60LbQtdK3HujKVwS", "name": "Ento" } } } ], "connections": {}, "pinData": {} } it returns

but without setting the type to JSON it returns

I did try leaving the header as just application/json but it returns this

As you suggested { "meta": { "instanceId": "7286cb7af5e69b909e9502fd59ea92cbd7d4a06c840dc2904f09626675230c35" }, "nodes": [ { "parameters": { "url": "=https://api.ento.com/au/api/shifts?filter[start_date]=2023-11-15&filter[end_date]=2023-11-15&filter[area_id]=76715&page[size]=25", "authentication": "genericCredentialType", "genericAuthType": "httpHeaderAuth", "sendHeaders": true, "headerParameters": { "parameters": [ { "name": "content-type", "value": "application/json" } ] }, "options": { "response": { "response": { "responseFormat": "json" } }, "pagination": { "pagination": { "paginationMode": "responseContainsNextURL", "nextURL": "=https://api.ento.com/au{{ JSON.parse($response.body.data).links. Next }}" } } } }, "id": "207e34ce-8469-4db6-bc0f-ccab43d11577", "name": "getShifts1", "type": "n8n-nodes-base.httpRequest", "typeVersion": 4.1, "position": [ 640, -20 ], "credentials": { "httpHeaderAuth": { "id": "60LbQtdK3HujKVwS", "name": "Ento" } } } ], "connections": {}, "pinData": {} }

Still returns