How to properly load test a web app?

[hartator]

Thanks for awesome work, help me considered Phoenix seriously.

I am building from scratch my own benchmarks and I was wondering what was your approach to load test with consistency?

Just trying to benchmark a simple nginx with nginx config tweaked for performance is already showing significant discrepancy:

➜  2.4.25 while true; do  wrk -t4 -c 100 -d30s http://0.0.0.0:8080 | grep "Requests/sec"; done 
Requests/sec:  54721.73
Requests/sec:  54753.20
Requests/sec:  37163.46
Requests/sec:  37637.49
Requests/sec:  37688.23
Requests/sec:  51645.86
Requests/sec:  23604.04
Requests/sec:  43615.19
Requests/sec:  32295.01
Requests/sec:  40335.15
Requests/sec:  37465.86
Requests/sec:  54761.35
Requests/sec:  37121.72
Requests/sec:  37446.36
Requests/sec:  36895.01
Requests/sec:  52580.09
Requests/sec:  22642.62

Moreover, adding more concurrency, it's resulting to random socket errors. I've tried to tweak ulimit -S -n xxx and launchctl maxfiles with limited success.

Running a similar config (iMac 27" - 32 gb of ram - i7 4.0ghz) on MacOS Sierra.

Any tips?