Run performance tests on different hardware environments

[heshanpadmasiri]

Currently we run our performance tests on a single type of EC2 instance. Instead we need to test on different kinds of instances to figure out their effects.

General purpose instances

VM	vCPU	Memory	Arch	Reason
t3.small	2	2GB	x86	Burstable instance
t4g.small	2	2 GB	ARM	Burstable instance
m7i-flex.xlarge	4	16 GB	x86	Claimed to have the best price performance among x86
m7g.xlarge	4	16 GB	ARM	Claimed to have the best price performance among ARM

Compute optimized

VM	vCPU	Memory	Arch	Reason
c4.xlarge	4	7.5	x86	Cheapest compute optimized
c7i.xlarge	4	8	x86	Highest x86 tier
c7gn.xlarge	4	8	ARM	Highest ARM tier

Memory optimized

VM	vCPU	Memory	Arch
r7i.large	2	16	x86

For each of these machines we are planning to test fallowing combinations

Constraint	Values (where applicable)
Heap size	1GB, 2GB, 4GB, 8GB
Concurrent users	100, 200, 500, 1000, 10000, 100000
XX:ActiveProcessorCount	1, 2, 4

Constraints

• We need atleast 1GB (this needs to be verified for ARM as well) heap size to run our tests with large message sizes, this means we need at least 2GB system memory

Objectives

• Figure out cases where we are CPU bound vs Memory bound
• Figure out our performance in ARM based systems (which tends to be cheaper than X86)
• Figure out our kneel points for each machine (ie. for a given machine at what point increasing number of users decrease the throughput (ie. we are bound by either CPU or Memory)