Add support for pagination

[stevegore]

The JIRA REST API applies pagination to a number of resources (e.g. users in group, issues returned from a search): https://docs.atlassian.com/jira/REST/server/#pagination

Ideally we would fetch the first page, and the fetch other pages concurrently.

Currently, it seems like this needs to be done manually with each implementation of the library.

[andygrunwald]

Sorry for the late reply @stevegore. You are right, sophisticated pagination support is not implemented (yet). At the moment you can reach this feature by applying the StartAt in your SearchOptions in your application code.

This can have pro and cons. Cons: Every app needs to implement this Pro: You might not flood your JIRA instance with requests.

Your idea to request this concurrent is quite nice. Thank you for this.

If someone would implement this and apply a PR, i would be happy to merge it. Right now i don't have time to implement it myself. Sorry. Would you mind to give it a try and add this feature?

[ghostsquad]

I would strong advice against automating any sort of "fetch all" functionality, as with the wrong query, this could easily timeout, or fetch more items that can be managed. Instead, I think we could implement a PaginatedResult type object, that simply makes it very easy to request the next page.

Some ideas for reference: https://github.com/qiangxue/golang-restful-starter-kit/blob/master/util/paginated_list.go

I didn't find much in the way of lazy evaluation in Go, but I suspect it's as simple as returning a function to the caller, which when invoked could fetch the next page.

func (p *PaginatedResult) Next() func() *PaginatedResult {
   ...
}

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity in the last 60 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[andygrunwald]

After some time, I am against my own comment from 2017. Pagination should be handled more explicitly.

Happy to accept PRs in this direction.

[fdcds]

At the moment you can reach this feature by applying the StartAt in your SearchOptions in your application code.

With GetIssuesForSprint I ran into the issue that I do not see a way to specify StartAt (#302).

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity in the last 60 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[eparis]

Since I ended up here I figured I'd post this for the next person implementing it in your code.

func GetIssues(client *jira.Client, searchString string) ([]jira.Issue, error) {
        last := 0
        var issues []jira.Issue = nil
        for {
                opt := &jira.SearchOptions{
                        MaxResults: 100,
                        StartAt:    last,
                }

                chunk, resp, err := client.Issue.Search(searchString, opt)
                if err != nil {
                        return nil, err
                }

                total := resp.Total
                if issues == nil {
                        issues = make([]jira.Issue, 0, total)
                }
                issues = append(issues, chunk...)
                last = resp.StartAt + len(chunk)
                if last >= total {
                        break
                }
        }
        return issues, nil
}

[andygrunwald]

Thanks Eric! Would you be interested in adding this code as an example via a Pull Request? This would be a great addition to the docs.

Eric Paris [email protected] schrieb am Mi. 19. Aug. 2020 um 21:59:

Since I ended up here I figured I'd post this for the next person implementing it in your code.

func GetIssues(client *jira.Client, searchString string) ([]jira.Issue, error) {

    last := 0

    var issues []jira.Issue = nil

    for {

            opt := &jira.SearchOptions{

                    MaxResults: 100,

                    StartAt:    last,

            }



            chunk, resp, err := client.Issue.Search(searchString, opt)

            if err != nil {

                    return nil, err

            }



            total := resp.Total

            if issues == nil {

                    issues = make([]jira.Issue, 0, total)

            }

            issues = append(issues, chunk...)

            last = resp.StartAt + len(chunk)

            if last >= total {

                    break

            }

    }

    return issues, nil

}

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/andygrunwald/go-jira/issues/55#issuecomment-676631140, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACOEQA7X2VLYESNQYK3YF3SBQVINANCNFSM4CZSHZMA .

[kp2401075]

Hey @andygrunwald & @eparis ,

I would be interested in adding @eparis's code as a pull request only if he is ok with it.

and if he is ok @andygrunwald would it go in issue.go?

[andygrunwald]

Hey @kp2401075, Adding this as an example would be a good idea. We aim to keep a fetch all functionality outside of this library. Pagination should be handled by the client/caller.

The library itself should cover attributes like StartAt to enable pagination. What do you think @kp2401075 ?

[kp2401075]

@andygrunwald That sounds good,

In that case where do you think should I add examples/jql ?

If somewhere else please suggest.

BTW, I'm very new to golang so If I do something stupid please point it out.

thanks.

[andygrunwald]

I suggest adding a new example with a particular focus on pagination rather than making an existing example more complex. Another idea would be to add a section to the readme

[eparis]

My code snippet above may be used in any context. It is public domain if your jurisdiction allows it. It may also be used under any open source license if a license is required. Go forth and use it!

[andygrunwald]

Hey,

I am very sorry that this issue has been open for a long time with no final solution. We work on this project in our spare time, and sometimes, other priorities take over. This is the typical open source dilemma.

However, there is news: We are kicking off v2 of this library 🚀

To provide visibility, we created the Road to v2 Milestone and calling for your feedback in https://github.com/andygrunwald/go-jira/issues/489

The development will take some time; however, I hope you can benefit from the changes. If you seek priority development for your issue + you like to sponsor it, please contact me.

What does this mean for my issue?

We will work on this issue indirectly. This means that during the development phase, we aim to tackle it. Maybe in a different way like it is currently handled. Please understand that this will take a while because we are running this in our spare time.

Final words

Thanks for using this library. If there is anything else you would like to tell us, let us know!