feat: add rate limiter

[dblythy]

New Pull Request Checklist

• [x] I am not disclosing a vulnerability.
• [x] I am creating this PR in reference to an issue.

Issue Description

Parse Server leaves rate limiting to the developer. As Parse is built for ease of use, internal rate limiting should be optional.

Related issue: #8170 Closes: #8170

Approach

Adds rate limiting via Parse Server config option:

rateLimitOptions: [
        {
          path: '/functions/*',
          windowMs: 10000,
          max: 1,
          message: 'Too many requests. Please try again later',
        },
      ],

However, I thought this still might be complex for new users who aren't sure how Parse Server mounts it's routes. So, I also added the ability to rate limit via a cloud validator:

Parse.Cloud.define('test', () => 'Abc', {
    rateLimit: {
    windowMs: 10000,
    max: 1,
    message: 'Too many requests. Please try again later',
  },
});

TODOs before merging

• [x] Add tests
• [x] A changelog entry is created automatically using the pull request title (do not manually add a changelog entry)

[parse-github-assistant[bot]]

Thanks for opening this pull request!

• 🎉 We are excited about your hands-on contribution!

[codecov[bot]]

Codecov Report

Base: 94.32% // Head: 87.41% // Decreases project coverage by -6.91% :warning:

Coverage data is based on head (56bcbe7) compared to base (a4990dc). Patch coverage: 95.53% of modified lines in pull request are covered.

:exclamation: Current head 56bcbe7 differs from pull request most recent head 160706c. Consider uploading reports for the commit 160706c to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##            alpha    #8174      +/-   ##
==========================================
- Coverage   94.32%   87.41%   -6.92%     
==========================================
  Files         181      181              
  Lines       14266    14357      +91     
==========================================
- Hits        13457    12550     -907     
- Misses        809     1807     +998     

Impacted Files	Coverage Δ
src/Options/index.js	100.00% <ø> (ø)
src/Routers/FilesRouter.js	87.07% <ø> (-5.45%)	:arrow_down:
src/middlewares.js	96.69% <93.44%> (-0.26%)	:arrow_down:
src/Config.js	90.32% <96.00%> (+0.49%)	:arrow_up:
src/GraphQL/ParseGraphQLServer.js	93.87% <100.00%> (+0.12%)	:arrow_up:
src/Options/Definitions.js	100.00% <100.00%> (ø)
src/ParseServer.js	91.85% <100.00%> (-0.32%)	:arrow_down:
src/cloud-code/Parse.Cloud.js	99.28% <100.00%> (+0.10%)	:arrow_up:
src/Adapters/Storage/Mongo/MongoCollection.js	6.97% <0.00%> (-90.70%)	:arrow_down:
src/Adapters/Files/GridFSBucketAdapter.js	10.14% <0.00%> (-84.06%)	:arrow_down:
... and 26 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

[dblythy]

Is it possible to specify different values for different routes

Yes, the option is an array and can specify specific routes if required.

rateLimitOptions: [
 {
    path: '/functions/*',
    windowMs: 100000,
    max: 50,
    message: 'Too many requests. Please try again later',
  },
  {
    path: '/functions/test',
    windowMs: 1000,
    max: 3,
    message: 'Rate limit specific to test,
  },
],

In this case all cloud functions will be restricted to 50 calls, whereas the function test will be restricted to 3.

[dblythy]

This also moves session lookup into its own middleware so that the rate limiter is performed before any database ops are ran

[dblythy]

Thank you for the thorough review @mtrezza :)

[dblythy]

I've just read that express-rate-limit doesn't support clusters either.

Perhaps rate limit flexible is a better option? Also supports using mongo/postgres as a backing store, and cluster mode.

Also has a block strategy against DDOS

[mtrezza]

I've just read that express-rate-limit doesn't support clusters either.

should also be added to the option docs

[dblythy]

Not really sure why the test to validate the config options is failing on Postgres with

Unhandled promise rejection: error: duplicate key value violates unique constraint "pg_class_relname_nsp_index" (line 534)

[mtrezza]

Thanks for the suggestion. This is not in scope for this PR. Please open a feature request after this feature has been introduced. Note that there is already a suggestion to switch to a different underlying rate limiter, so exposing the rate limiter directly to the developer is (at least at this stage) not desirable.

[Moumouls]

I've just read that express-rate-limit doesn't support clusters either.

Perhaps rate limit flexible is a better option? Also supports using mongo/postgres as a backing store, and cluster mode.

Also has a block strategy against DDOS

https://github.com/nfriedly/express-rate-limit support cluster, store options available (redis, memcached, etc...)

Note that there is already a https://github.com/parse-community/parse-server/pull/8174#issuecomment-1252360142 to switch to a different underlying rate limiter

No need to switch @mtrezza the current one support some stores.

Here we just need to expose the rate limiter option in Parse Config options. Easy to add, why we should not support it, since it just need to forward an options object through some functions?

I'm also sure that exposing the options is REQUIRED because some developers will NEED to disable/adjust settings depending on the proxy in front of parse like

{
	windowMs: 15 * 60 * 1000, // 15 minutes
	max: 100, // Limit each IP to 100 requests per `window` (here, per 15 minutes)
	standardHeaders: true, // Return rate limit info in the `RateLimit-*` headers
	legacyHeaders: false, // Disable the `X-RateLimit-*` headers
}

[mtrezza]

This requires further discussion which is beyond the scope of this PR. It may be easy to expose options, but the discussions around it require more deliberation.

@dblythy Is https://github.com/parse-community/parse-server/pull/8174#issuecomment-1252461985 the issue left to merge this? I restarted the CI, maybe some tests are just flaky. If you are currently only testing with MongoDB, you could spin up a Postgres instance (see contribution guide for how to do that easily) and run the test locally, then see where it throws.

[mtrezza]

Does this rate limiter also work for GraphQL API request and file requests? It should work for file requests, but GraphQL requests are out of scope of this PR.

[mtrezza]

@Moumouls you've made some good remarks. It would be great if you could open an issue so we track them. We'll keep this PR small, so we can merge a basic version and hopefully extend it further with your ideas.

[stewones]

👀 very impressive work guys, can't wait for this feature!

[parse-github-assistant[bot]]

I will reformat the title to use the proper commit message syntax.

[dblythy]

Future evolution:

• add more options to rate limit options
• add redis memory store to allow for clusters / load balanced environments
• add "zone" where rate limit can be restricted by sessionToken (easy to implement, inexpensive) or user Id (requires a DB operation), or IP

[mtrezza]

Is this ready for review or still WIP?

[parseplatformorg]

🎉 This change has been released in version 6.0.0-alpha.21

[cbaker6]

@dblythy I attempted to enable the rate limiter, but I couldn't get it to allow any connections. Definitely possible I'm configuring or understanding how to use it incorrectly, though the server seems to like my configuration. I didn't want to open an issue yet, as it could be a config problem on my side. I attempted to try similar configurations to your tests. Below is what I try (you can see the whole file here:

// Use default rate limiter values for PARSE_SERVER_RATE_LIMIT_REQUEST_PATH = '*'
// Configure Parse Server using startApp
const parseServer = await ParseServer.startApp(configuration);

Have you gotten this to work on docker or another setup outside of the test environment? If so, can you share relevant parts of your config?

[mtrezza]

@cbaker6 Please use the community forum or slack for code level support to prevent fragmented discussions.

[dblythy]

@cbaker6 are you saying that the rate limiter is rejecting all requests? It could possibly be related to trust proxy:

Troubleshooting Proxy Issues

If you are behind a proxy/load balancer (usually the case with most hosting services, e.g. Heroku, Bluemix, AWS ELB, Nginx, Cloudflare, Akamai, Fastly, Firebase Hosting, Rackspace LB, Riverbed Stingray, etc.), the IP address of the request might be the IP of the load balancer/reverse proxy (making the rate limiter effectively a global one and blocking all requests once the limit is reached) or undefined. To solve this issue, add the following line to your code (right after you create the express application):

app.set('trust proxy', numberOfProxies) Where numberOfProxies is the number of proxies between the user and the server. To find the correct number, create a test endpoint that returns the client IP:

app.set('trust proxy', 1) app.get('/ip', (request, response) => response.send(request.ip)) Go to /ip and see the IP address returned in the response. If it matches your public IP address, then the number of proxies is correct and the rate limiter should now work correctly. If not, then keep increasing the number until it does.

https://github.com/express-rate-limit/express-rate-limit#troubleshooting-proxy-issues

[dblythy]

@cbaker6 I'm thinking you may have observed #8399

[cbaker6]

Definitely possible, I'll test your new PR once it's merged and let you know if it fixed the problem

[parseplatformorg]

🎉 This change has been released in version 6.0.0-beta.1

[parseplatformorg]

🎉 This change has been released in version 6.0.0