autopush-rs icon indicating copy to clipboard operation
autopush-rs copied to clipboard

feat: Load test fixups

Open jrconlin opened this issue 1 year ago • 8 comments

Some minor tweaks to the load tests to make them easier to run locally.

These include:

  • normalizing the --config option across binaries (this is not used in production)
  • Change the load test to use a hash of the payload instead of the payload itself (freeing memory and making checks faster)
  • Make the WORKER_COUNT an environment parameter

jrconlin avatar Jan 29 '24 23:01 jrconlin

I ran this locally and was getting some errors. image

@Trinaa says that this is from a race condition that comes from using dict's as the data type for storing the records. I want to look into this a bit so I can understand what is happening.

b4handjr avatar Jan 31 '24 23:01 b4handjr

I guess I'm not running the load tests correctly, because I'm not seeing those failures: image

(This is with both the load tests and servers running on my local machine. There are a bunch of connection refused errors, but those are to be expected.)

boosting the local ulimit to 2048 also helped with some of the file handle problems.

jrconlin avatar Feb 01 '24 17:02 jrconlin

I guess I'm not running the load tests correctly, because I'm not seeing those failures...

@jrconlin the list of request names looks a bit short in your screenshot. There should be some POST notification - send and WSS notification - recv in there...

Trinaa avatar Feb 01 '24 21:02 Trinaa

Yeah, I realized that I was running low on node handles. I ran another test now that I've kicked the count up to stupid levels. Screenshot 2024-02-01 at 16-35-57 Locust for None

The 1 failure was a database save error resulting in a 503

jrconlin avatar Feb 02 '24 00:02 jrconlin

Yeah, I realized that I was running low on node handles. I ran another test now that I've kicked the count up to stupid levels. Screenshot 2024-02-01 at 16-35-57 Locust for None

The 1 failure was a database save error resulting in a 503

Can you try running against the stage server? The URL should be set by default.

b4handjr avatar Feb 02 '24 16:02 b4handjr

I can, the reason I was running locally was to load test dual mode. (I had configured autopush to run as that locally as well) I'm presuming that stage is still configured to run against the default dynamodb, which is fine, since we can get a good baseline that way. I'll work with @emaydeck-mozilla about what the dual configuration should be.

jrconlin avatar Feb 02 '24 16:02 jrconlin

@jrconlin Do we want to merge this still?

b4handjr avatar Mar 06 '24 21:03 b4handjr

If it's still relevant and useful. yes.

jrconlin avatar Mar 12 '24 19:03 jrconlin