[HTTP Json Chain Call] Fix Pagination not working at the root request level when used with chaining

[vinit-chauhan]

Describe the enhancement: While using the chain call in HTTP JSON input, we cannot use the body of the first call. Also, in pagination, the chain call is not working, only giving the response from the first request, the second call is not happening.

Describe an enhancement for Pagination in chain call: we came across an issue while using the chained calls feature that is introduced in filebeat recently. In our case, we wanted to make an API call that would fetch a paginated list of IDs and we need to iterate through those IDs in order to get their details. However, we were not able to use pagination on chained calls. We are unable to access the body of the first page.

**Example**

First call
{
	items:[
	      Id: 1
              Id: 2
              Id: 3
              Id: 4
              Id: 5]	
        nextLink: “”
}
Chain Call
           Second Call Output_id-1
           Second Call Output_id-2
           Second Call Output_id-3
           Second Call Output_id-4
           Second Call output_id-5

In this example, the first call collects all the ids and the second call gives the detail of all the ids. And for the pagination, we have to use the nextLink, that is present in the body of the first call.

In our case, for pagination, we have to use the nextLink that is present in the body of the first call. We saw that there is no way to use the body of the first call. We need to access the body of the first call.

If we are using the last_response element, we shaw it will give the response of the first id of the second call(Output_id-1). In this scenario, we have tested the mock server.

In pagination, we shaw that it will call every time only the first call, the second call did not happen in this pagination section.

Moreover, Pagination and Chained calls are working fine individually, but it fails when combined. we are not able to use the body of the first call in pagination. And also when we found that in pagination the chain request is not happening, every they call the first call and give the response of list of IDs.

Found while investigating, we saw that in the https://github.com/elastic/beats/blob/main/x-pack/filebeat/input/httpjson/pagination.go#L141, they iterate the first request every time. In this function, we have to call chain requests in this function.

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[ShourieG]

@vinit-elastic The pagination not working when placed at the root request level when used with chaining is a genuine bug and we will take it up as a fix. However for nested chain calls to also support pagination is a new feature and should have a separate ticket of its own. This ticket should be renamed appropriately to a bugfix ticket .

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)