Support rate limiting

[alexanderkjeldaas]

Most APIs have some sort of rate-limit.

This rate-limit can be per API call (for example AWS does this), or for the whole API.

It might not be reasonable to encode this in the types directly, but some way of configuring this would make sense, at least as a global rate-limit as wrapping every function is tedious boiler-plate.

[dmjio]

I believe this is available in the form of wai middleware, https://hackage.haskell.org/package/wai-middleware-throttle-0.2.0.2/docs/Network-Wai-Middleware-Throttle.html

You should be able to plug this in where servant-server generates an Application type, so something like this.

main = do
  throttler <- initThrottler
  let f = throttle defaultThrottleSettings throttler
  let app = Servant.Server.serve (Proxy :: Proxy API) handlers
  Warp.run 3000 (f app)

Should be configurable via the throttle settings, and would provide global rate-limiting.

[alpmestan]

A global or more fine-grained rate-limiting can be implemented as a WAI middleware. As a matter of fact, there's a library on hackage that provides precisely this: wai-middleware-throttle.

Applying a middleware mw :: Middleware to a servant app is as simple as applying mw (which is just a function) to the result of Servant.Server.serve, as you can see in this example.

I'm not sure one would get a lot of mileage out of being able to specify rate limiting in API descriptions, as this information is very much server-specific and the other interpretations wouldn't really get anything out of it. If you disagree, I'd be curious to hear about it!

[alexanderkjeldaas]

Sorry, this issue should be on servant-client, not the serving part.

[jkarni]

@alexanderkjeldaas are the middleware options sufficient? If so, can we close this?

(I also don't quite understand your last comment. Do you mean we should have servant-client do leg work for rate-limiting? If so, how would that work?)

[alexanderkjeldaas]

Yes that's what I mean. All useful APIs have rate-limits, so someone has to build this support.

On Thu, Dec 10, 2015 at 1:25 AM, Julian Arni [email protected] wrote:

@alexanderkjeldaas https://github.com/alexanderkjeldaas are the middleware options sufficient? If so, can we close this?

(I also don't quite understand your last comment. Do you mean we should have servant-client do leg work for rate-limiting? If so, how would that work?)

— Reply to this email directly or view it on GitHub https://github.com/haskell-servant/servant/issues/266#issuecomment-163446009 .

[ehamberg]

It would be great to have rate-limiting, but I guess that belongs in the http-client/http-client-tls libraries that servant-client uses?

[jkarni]

OK, I see. Given that rate-limiting would show up (a) in the server, (b) in the documentation, and (c) in clients, I see the advantage of having a combinator rather than using just middleware. That said, I don't quite know how we'd support rate-limiting in the client. http-client doesn't seem to have support for it.

EDIT: @ehamberg I guess it belongs in those libraries first, even if not only.

[jkarni]

This will get a lot easier with #327 for servant-server.

[ghost]

Hello, I have started to do some basic work on this (https://github.com/sachs4/servant/tree/rate-limit). It still has a lot of warts, but if somebody could take a short look over it and could say if the general direction of this change is good I would try to make it better.

[jkarni]

@sachs4 nice. I have a couple of questions or comments:

1. I think our experience has been that adding a single combinator with more configuration options (and then possibly type synonyms) is better than having multiple combinators. Thus, instead of RateLimitHeader, RateLimitIP, etc., RateLimit ByHeader, RateLimit ByIp, etc. This saves us a lot of instances, especially in packages that treat the configuration option uniformly.

2. The HasPathToCombinator is very interesting, even outside of the rate-limiting context. However, the changes are pretty extensive, and (as far as I can tell) there are questions about whether it'd be witnessing distinctions between APIs that we would like to treat equivalently, and moreover it overlaps somewhat with the functionality RouterStructure. It would take a lot of discussion, and quite convincing use cases, for this change to be merged. A much less intrusive approach (which may have issues - I haven't thought it through extensively) might be something like:

data RateLimit (limitId :: Symbol) limitType limitAmount
-- e.g. `RateLimit "repo-api" ByIP (100 `Per` Minute)

And then any sub-apis under (to the left of :> of) a particular RateLimit combinator would add to the count named in the combinator. In addition to probably simplifying the implementation, this has the advantage, it seems to me, of allowing join limits over sub-APIs that don't adjoin one another.

3. I'd have to think more about the data structure containing the counts. I'm inclined to think that we should be able to not have contention on Thingys. On the other hand, that might not be a worthwhile effort since, as I understand it, that basically corresponds to an API user. There are some existing libraries that deal directly with rate limiting (e.g. rate-limit) that might be interesting to look at. (I'm not saying that's a better way to go - just an alternative.) For the inner map as well, using a PSQueue instead might be worth thinking about.

Thanks a lot!

[soenkehahn]

Hey @sachs4,

thanks for looking into this. I just took a quick look at your PR. Tbh, I wouldn't want to merge this for the following reasons:

• We already have a wai Middleware for rate limiting on the server side. On the client side this should ideally be in http-client (and then maybe servant-client could piggy-back on that). So it feels to me that a rate limiting combinator is only useful in a quite small number of cases.
• I don't want to see the HasServer class change for this.
• I'm not sure whether I want to see Delayed change for this. This is a much less public interface than HasServer though, so it might be ok.
• Is there any way to put this in a separate package? I think this would be much better, because it would put the maintenance burden on people that actually care about the feature.

The last point is problematic, since we don't have a good way of letting people write combinators and just put them in separate packages on github and hackage, due to dependency blow-up. @sachs4: But maybe you can manage?

Sorry, I don't have time to write about this problem in depth atm, but I didn't want to hold back on feedback for this PR.

[ghost]

@jkarni, @soenkehahn thank you for your kind feedback. I've started to fix some of your points. The combinators are now combined into one single. The HasPathToCombinator is gone. The changes to the HasServer class have been reverted. I'm starting to rework @jkarni's 3rd point. While writing some test I ran into this issue https://github.com/CRogers/should-not-typecheck/issues/5, @jkarni do you know any fix for this yet? Regarding servant-docs, would the ability to create charts like this https://dev.twitter.com/rest/public/rate-limits be desirable? Is there any specific Markdown standard the servant-docs package should follow?

[jkarni]

Great news!

Re: should-not-typecheck. I don't know any fix, but you can use doctest to verify type errors exist and say what you expect. I've been thinking about moving away from should-not-typecheck to doctest because of that issue.

Re: charts. That would definitely be very cool, but I imagine it'd make sense to separate that out as a later PR?

[osa1]

Anyone have any progress on this? Applying wai-middleware-throttle to the whole Application is not always possible (because we don't want to throttle all endpoints), and I don't want to use a Raw endpoint just to be able to throttle. I looked at the links above, but it seems like servant types are quite different now, so they're not easily usable.

[domenkozar]

There are a couple of solutions here, I'll very briefly mention them here according to my research:

a) Do the rate limiting before hitting servant. I have a preference for this because I don't think this should be a job of the application.

b) Do the rate limiting with wai middleware.

c) Do the rate limiting per servant route.

1. If you want to limit concurrent requests, use something like bracket_ waitQSem signalQSem body using https://hackage.haskell.org/package/base-4.12.0.0/docs/Control-Concurrent-QSem.html
2. If you want to rate limit the requests and ease out the spikes, see https://hackage.haskell.org/package/token-limiter-concurrent

This mostly needs a cookbook recipe to be written.

[mbg]

I have just released servant-rate-limit which allows applying different (or no) rate limiting strategies for different endpoints. For example:

import Servant
import Servant.RateLimit

type TestAPI
    = RateLimit (FixedWindow 2 50) (IPAddressPolicy "fixed:") :>
      "fixed-window" :>
      Get '[JSON] String
 :<|> RateLimit (SlidingWindow 2 50) (IPAddressPolicy "sliding:") :>
      "sliding-window" :>
      Get '[JSON] String
 :<|> "unrestricted" :>
      Get '[JSON] String

See the tests for a full code example and the README for an explanation.

This library builds on top of my wai-rate-limit library which should support different backends. Currently there is only a Redis backend. Here's an example for how to initialise this:

testApi :: Proxy TestAPI
testApi = Proxy

main :: IO ()
main = do
    -- connect to the Redis server and construct a backend for the connection
    backend <- redisBackend <$> checkedConnect defaultConnectInfo

    -- stick the Redis backend into the Servant context so that we can access
    -- it when we try to apply rate limiting policies
    let ctx = backend :. EmptyContext

    -- construct the Servant application using the context
    let app = serveWithContext testApi ctx server

It should be relatively straight-forward to implement custom strategies (how to apply rate limiting) and policies (how to identify who to apply them to). For example, a custom policy which uses a username from the request vault as identifier (this could e.g. replace IPAddressPolicy in TestAPI):

import qualified Data.Vault.Lazy as V

type UserId = ByteString -- for simplicity

{-# NOINLINE userKey #-}
userKey :: V.Key UserId
userKey = unsafePerformIO newKey

data MyPolicy

instance HasRateLimitPolicy MyPolicy where
    type RateLimitPolicyKey MyPolicy = ByteString

    policyGetIdentifier req =
        fromMaybe (error "expected to have a user id in the vault") $
        V.lookup userKey (vault req)