nginx-gateway-fabric icon indicating copy to clipboard operation
nginx-gateway-fabric copied to clipboard

Published 20 hours ago • verified publisher icon nginxinc

Automate scale test

Open pleshakov opened this issue 1 year ago • 1 comments
trafficstars

Proposed changes

Problem: Non-functional scale test needs to be run manually.

Solution:

  • Automate scale test.
  • Use in-cluster Prometheus to collect CPU, memory and NGF metrics.
  • Use Kubernetes API server to get NGF logs.

For development and troubleshooting, it is possible to run scale test locally in Kind cluster. However, it is necessary to bring down the number of HTTPRoutes to 50 or less (roughly).

Testing:

  • Ran this test locally with 64 listeners, 50 routes and 50 upstreams with NGINX OSS.
  • Ran this test on GKE with the default configuration with NGINX OSS. Test results -- https://github.com/pleshakov/nginx-gateway-fabric/commit/55be967bfa6d6b957e46249506aadb9bea637edc

Out of scope: ensuring this test runs successfully via GitHub pipeline -- https://github.com/nginxinc/nginx-gateway-fabric/issues/1927

Closes https://github.com/nginxinc/nginx-gateway-fabric/issues/1368

Largely based on work by Ciara in https://github.com/nginxinc/nginx-gateway-fabric/pull/1804

Checklist

Before creating a PR, run through this checklist and mark each as complete.

  • [x] I have read the CONTRIBUTING doc
  • [x] I have added tests that prove my fix is effective or that my feature works
  • [x] I have checked that all unit tests pass after adding my changes
  • [x] I have updated necessary documentation
  • [x] I have rebased my branch onto main
  • [x] I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

Release notes

If this PR introduces a change that affects users and needs to be mentioned in the release notes, please add a brief note that summarizes the change.

NONE

pleshakov avatar May 02 '24 19:05 pleshakov

If you wanted to verify that this test can be run in the pipeline and close out https://github.com/nginxinc/nginx-gateway-fabric/issues/1927, you could create this branch on the upstream repo and then run the workflow on that branch.

sjberman avatar May 16 '24 20:05 sjberman

If you wanted to verify that this test can be run in the pipeline and close out https://github.com/nginxinc/nginx-gateway-fabric/issues/1927, you could create this branch on the upstream repo and then run the workflow on that branch.

@sjberman it needs a bigger cluster, that's why I'd rather not update it here

pleshakov avatar May 16 '24 21:05 pleshakov