Benchmark the graphql-api service

[sjahl]

I'd like to understand our base-line performance with a variety of gnomAD backend topologies.

[sjahl]

I've deployed a testing gnomad-browser in gnomadev, with the following changes to the elasticsearch cluster:

• switched from a 60:1 ratio of "GB of disk" to "GB of JVM Heap" to a 30:1 ratio.
  • This was done by halving the size of the disks attached to elasticsearch pods, and doubling the number of pods, with the same memory/CPU requests and limits
• Added one replica to each index. This again doubles the size of the ES cluster to accommodate the extra disk requirements

The result is a 9-node ES cluster (one extra node to ensure I didn't run out of disk space.

The plan is to run JMeter against the API on both this new test cluster and the soon-to-be prod deployment that mirrors the current production 2-node ES cluster topology.

[sjahl]

The results of my load testing are stashed in gs://gnomad-jmeter-reports. Some of the tests have a "cold" and "warm" denotation -- Cold is cases where JMeter is being run for the first time against a cluster that has only had the cache warmer script run on it, which I discovered didn't really warm things up to the point that I would consider the cluster adequately cached.

I will compile the results here into a comparison and review the findings for the team.