DependencyTrack
Consider using cursor- or keyset-based pagination
Current Behavior
As a DT instance continues to grow, paginating through REST API resources becomes increasingly more expensive.
This is because we leverage offset-based pagination everywhere. The problem with this approach is that the database ends up having to load records that it immediately discards again.
Cursor- or keyset-based pagination can avoid this:
- https://use-the-index-luke.com/no-offset
- https://use-the-index-luke.com/sql/partial-results/fetch-next-page
- https://www.cockroachlabs.com/docs/stable/pagination
- https://readyset.io/blog/optimizing-sql-pagination-in-postgres
- https://www.citusdata.com/blog/2016/03/30/five-ways-to-paginate/
It is further questionable whether the current offset-based pagination provides any value to end-users. If a result set contains 1000s of records, offering an option to jump to page 666 of 1000 serves no purpose.
What helps users get to where they want to, is more filtering options to narrow down the result set to, ideally, only a small handful of records.
Proposed Behavior
Investigate the feasibility of leveraging cursor-, or keyset-based pagination for REST resources.
A limitation of these approaches is that user-defined ordering is severely restrained.
It is likely that the lack of custom ordering abilities can be compensated with more flexible filtering methods.
For example, instead of ordering a list of vulnerabilities by the number of projects they affect, provide a means of filtering results to only return vulnerabilities affecting between X and Z projects.
Checklist
- [x] I have read and understand the contributing guidelines
- [x] I have checked the existing issues for whether this enhancement was already requested
Marked as size/M, because implementing this for a small subset of REST endpoints to begin with should be possible to do with reasonable effort.
The biggest part of all this would be research, and finally coming up with something that can be utilized across the application in a consistent manner.