Properly Handle Overwhelmed Search Endpoints like APIs that return 429 HTTP Code

[epugh]

Is your feature request related to a problem? Please describe. We have seen with some search endpoints, especially those that are APIs (not actual Solr/OS/ES endpoints) that if we queue up thousands of queries, we overwhelm the API, and get back HTTP 429 errors. Quepid doesn't understand what a HTTP 429 return code is, or how to handle it, instead it just treats it as a failed request.

I believe that the key bit of code is https://github.com/o19s/quepid/blob/main/app/assets/javascripts/services/queriesSvc.js#L802 where we have a return $q.all(promises) and the promises are all the individual search requests that we've queued up: promise = query.search().

Describe the solution you'd like We really want to handle the 429 error code. To be properly robust, we want to see a 429, and retry after some period. It would be especially cool if we can look at the header like Retry-After: 5 and incorporate that delay.

Describe alternatives you've considered

1. I have tried some Angular 1 libraries that purport to throttle the $http object, like https://github.com/mikepugh/angular-http-throttler and https://github.com/bhollis/ng-http-rate-limiter.

2. I also thought, hey, can we route the traffic through Quepid using the ProxyController that we already have, and then thottling the traffic there using something like https://github.com/rack/rack-attack?tab=readme-ov-file#throttling. However, the problem is that what we want is to queue up 1000 requests, and then forward 5 at a time. But the rack-attack appears to instead return 429s. Now, if Quepid understood 429, then the proxy could be a way of establishing a limit for API's that don't natively support that.

Additional context I have a branch out, https://github.com/o19s/quepid/pull/1070 but nothing really working. https://github.com/o19s/quepid/pull/1070/files#diff-99826bf177a5ca680203979c9576cd10ceb3e0dcc6b809e8bc2c96dc15568927 is some examples of different ways of running the searchAll method in the queriesSvc.js.