drill
drill copied to clipboard
fcsonline
Poor performance
config:
threads: 16
base: 'http://localhost:8080'
iterations: 1000
rampup: 2
plan:
- name: json
request:
url: /json
result:
./target/release/drill --benchmark benchmark.yml --stats -q
Threads 16
Iterations 1000
Rampup 2
Base URL http://localhost:8080
json Total requests 16000
json Successful requests 16000
json Failed requests 0
json Median time per request 3ms
json Average time per request 5ms
json Sample standard deviation 5ms
Concurrency Level 16
Time taken for tests 21.4 seconds
Total requests 16000
Successful requests 16000
Failed requests 0
Requests per second 746.78 [#/sec]
Median time per request 3ms
Average time per request 5ms
Sample standard deviation 5ms
run same benchmark with ab:
ab -n 16000 -k -c 16 http://localhost:8080/json
This is ApacheBench, Version 2.3 <$Revision: 1807734 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 1600 requests
Completed 3200 requests
Completed 4800 requests
Completed 6400 requests
Completed 8000 requests
Completed 9600 requests
Completed 11200 requests
Completed 12800 requests
Completed 14400 requests
Completed 16000 requests
Finished 16000 requests
Server Software: wizzardo
Server Hostname: localhost
Server Port: 8080
Document Path: /json
Document Length: 27 bytes
Concurrency Level: 16
Time taken for tests: 0.497 seconds
Complete requests: 16000
Failed requests: 0
Keep-Alive requests: 16000
Total transferred: 2832000 bytes
HTML transferred: 432000 bytes
Requests per second: 32195.81 [#/sec] (mean)
Time per request: 0.497 [ms] (mean)
Time per request: 0.031 [ms] (mean, across all concurrent requests)
Transfer rate: 5565.10 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 2
Processing: 0 0 1.6 0 41
Waiting: 0 0 1.5 0 41
Total: 0 0 1.6 0 41
Percentage of the requests served within a certain time (ms)
50% 0
66% 0
75% 0
80% 1
90% 1
95% 2
98% 4
99% 6
100% 41 (longest request)
Thanks for you report. Some users compared ab with drill. I'm going to try to explain what is going on here. Although drill and ab are tools that can be compared like you did there is small difference that make this comparation inaccurate.
ab is targeting a single URL (http://localhost:8080/json) and run all threads against it. They can do async requests maximizing the throughput. drill executes a test plan iteration sequentially per each thread, because you can have dependencies between steps like you can see in the documentation. This forces drill to wait for the request body, to parse it and give it to next step as available content.
Although these examples look the same in terms of throughput (16 * 500 * 2), all Second json requests done indrill are going to wait to First json in each iteration. Example:
Ab:
ab -n 16000 -k -c 16 http://localhost:8080/json
Drill:
threads: 16
base: 'http://localhost:8080'
iterations: 500
plan:
- name: First json
request:
url: /json
- name: Second json
request:
url: /json
Also, be careful with rampup setting because if slowing down a bit the test plan. Just remove it.
I'm sure there are some improvements to be done and are quite related with asynchronous version of Hyper with Tokio promises to request them independently from the test plan if you know there are no dependencies between them.
Does this make sense?
Also, I can see that you are using the keep-alive flag in ab. Right now, drill doesn't support it. This could add an extra delay for each request connection initialization.
ab is uses only one thread, but sends requests in multiple connections, every connection waits for a response before sending another requests, so it's the same synchronous logic that drill has. Async requests - http pipelining, ab doesn't support it, wrk can do them and it reaches 3kk rps on my laptop. Also I don't know any browser or http-client that doesn't have keep-alive nowdays, if you want to support http/1.1 you have to implement it
./target/release/drill --benchmark benchmark.yml --stats -q
Invalid rampup value!
Threads 1
Iterations 10000
Rampup 0
Base URL http://localhost:8080
json Total requests 10000
json Successful requests 10000
json Failed requests 0
json Median time per request 0ms
json Average time per request 0ms
json Sample standard deviation 0ms
Concurrency Level 1
Time taken for tests 49.6 seconds
Total requests 10000
Successful requests 10000
Failed requests 0
Requests per second 201.79 [#/sec]
Median time per request 0ms
Average time per request 0ms
Sample standard deviation 0ms
ab -n 10000 -c 1 http://localhost:8080/json
This is ApacheBench, Version 2.3 <$Revision: 1807734 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests
Server Software: wizzardo
Server Hostname: localhost
Server Port: 8080
Document Path: /json
Document Length: 27 bytes
Concurrency Level: 1
Time taken for tests: 0.667 seconds
Complete requests: 10000
Failed requests: 0
Total transferred: 1530000 bytes
HTML transferred: 270000 bytes
Requests per second: 14986.68 [#/sec] (mean)
Time per request: 0.067 [ms] (mean)
Time per request: 0.067 [ms] (mean, across all concurrent requests)
Transfer rate: 2239.22 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 0 0 0.0 0 1
Waiting: 0 0 0.0 0 1
Total: 0 0 0.0 0 1
Percentage of the requests served within a certain time (ms)
50% 0
66% 0
75% 0
80% 0
90% 0
95% 0
98% 0
99% 0
100% 1 (longest request)
I merged some commits that improve the performance. I found that SSL connector was adding an important delay. Could you pull from master and try it again?
much better!
wizzardo@xps:~/projects/drill$ ./target/release/drill --benchmark benchmark.yml --stats -q
Threads 1
Iterations 100000
Rampup 0
Base URL http://localhost:8080
json Total requests 100000
json Successful requests 100000
json Failed requests 0
json Median time per request 0ms
json Average time per request 0ms
json Sample standard deviation 0ms
Concurrency Level 1
Time taken for tests 13.0 seconds
Total requests 100000
Successful requests 100000
Failed requests 0
Requests per second 7696.66 [#/sec]
Median time per request 0ms
Average time per request 0ms
Sample standard deviation 0ms
but still slower than ab
ab -n 100000 -c 1 http://localhost:8080/json
This is ApacheBench, Version 2.3 <$Revision: 1807734 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests
Server Software: wizzardo
Server Hostname: localhost
Server Port: 8080
Document Path: /json
Document Length: 27 bytes
Concurrency Level: 1
Time taken for tests: 5.821 seconds
Complete requests: 100000
Failed requests: 0
Total transferred: 15300000 bytes
HTML transferred: 2700000 bytes
Requests per second: 17179.15 [#/sec] (mean)
Time per request: 0.058 [ms] (mean)
Time per request: 0.058 [ms] (mean, across all concurrent requests)
Transfer rate: 2566.81 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 0 0 0.1 0 11
Waiting: 0 0 0.1 0 11
Total: 0 0 0.1 0 11
Percentage of the requests served within a certain time (ms)
50% 0
66% 0
75% 0
80% 0
90% 0
95% 0
98% 0
99% 0
100% 11 (longest request)
Could you checkout feature/slow branch and test it with 1 iteration, please? This is going to show where the time is spent during a request.
Step1: 0.0000005399924702942371 Step2: 0.000005211972165852785 Step3: 0.000008183007594197989 Step4: 0.000010876974556595087 Step5: 0.000018504972103983164 Step6: 0.000021085026673972607 Step7: 0.000023559026885777712
Step1: 0.0000005509937182068825 Step2: 0.000005176989361643791 Step3: 0.000008131959475576878 Step4: 0.000010748975910246372 Step5: 0.0000180340139195323 Step6: 0.000020682986360043287 Step7: 0.000023145985323935747
Step1: 0.0000005399924702942371 Step2: 0.000005270005203783512 Step3: 0.00000824901508167386 Step4: 0.000010951014701277018 Step5: 0.000018224993254989386 Step6: 0.0000208010314963758 Step7: 0.000023383006919175386
Step1: 0.0000004820176400244236 Step2: 0.000005076988600194454 Step3: 0.000007946975529193878 Step4: 0.000010665971785783768 Step5: 0.000017902988474816084 Step6: 0.000020487001165747643 Step7: 0.00002230401150882244
Hey @fcsonline 🖖
I am also having issues with drill's performance. Just to put in context, I benchmarked an endpoint that only returns "OK".
Fetch games Total requests 5000
Fetch games Successful requests 5000
Fetch games Failed requests 0
Fetch games Median time per request 2ms
Fetch games Average time per request 37ms
Fetch games Sample standard deviation 256ms
Concurrency Level 5
Time taken for tests 37.4 seconds
Total requests 5000
Successful requests 5000
Failed requests 0
Requests per second 133.65 [#/sec]
Median time per request 2ms
Average time per request 37ms
Sample standard deviation 256ms
for which ab yields
Server Hostname: 0.0.0.0
Server Port: 3000
Document Path: /_check2
Document Length: 2 bytes
Concurrency Level: 5
Time taken for tests: 0.284 seconds
Complete requests: 5000
Failed requests: 0
Total transferred: 590000 bytes
HTML transferred: 10000 bytes
Requests per second: 17582.79 [#/sec] (mean)
Time per request: 0.284 [ms] (mean)
Time per request: 0.057 [ms] (mean, across all concurrent requests)
Transfer rate: 2026.14 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 1
Processing: 0 0 0.0 0 1
Waiting: 0 0 0.0 0 1
Total: 0 0 0.0 0 1
Percentage of the requests served within a certain time (ms)
50% 0
66% 0
75% 0
80% 0
90% 0
95% 0
98% 0
99% 0
100% 1 (longest request)
I have been working lately on a branch to introduce async/await futures in drill. This will bring better performance for the use case you mentioned. You can review it here:
https://github.com/fcsonline/drill/pull/33
It's not ready yet, but I'm going to try to merge as soon as possible.
Any updates on this issue? Our team loves drill so thank you for this project! We really want to incorporate it as our main loadtest framework if performance keeps increasing as it has throughout this thread.
didn't try the pr, but master already looks good:
./target/release/drill -q --benchmark benchmark.yml --stats
Threads 4
Iterations 10000
Rampup 2
Base URL http://localhost:8080
Fetch json example Total requests 40000
Fetch json example Successful requests 40000
Fetch json example Failed requests 0
Fetch json example Median time per request 0ms
Fetch json example Average time per request 0ms
Fetch json example Sample standard deviation 1ms
Concurrency Level 4
Time taken for tests 2.4 seconds
Total requests 40000
Successful requests 40000
Failed requests 0
Requests per second 16860.41 [#/sec]
Median time per request 0ms
Average time per request 0ms
Sample standard deviation 1ms
but still waiting for keep-alive, my test app can handle more than 100k rps
@wizzardo Can you run the same test without rampup? This period of time is counted in Time taken for tests and Requests per second.
Also, it could be cool to test the async branch. #46
About the keep-alive think, I will start working on this after merge the async branch of @messense
master branch
time ./target/release/drill --benchmark benchmark.yml --stats -q
Threads 4
Iterations 100000
Rampup 0
Base URL http://localhost:8080
Fetch example json Total requests 400000
Fetch example json Successful requests 400000
Fetch example json Failed requests 0
Fetch example json Median time per request 0ms
Fetch example json Average time per request 0ms
Fetch example json Sample standard deviation 1ms
Concurrency Level 4
Time taken for tests 26.3 seconds
Total requests 400000
Successful requests 400000
Failed requests 0
Requests per second 15181.49 [#/sec]
Median time per request 0ms
Average time per request 0ms
Sample standard deviation 1ms
real 0m26.553s
user 0m15.054s
sys 1m18.425s
async pr
time ./target/release/drill --benchmark benchmark.yml --stats -q
Threads 4
Iterations 100000
Rampup 0
Base URL http://localhost:8080
Fetch example json Total requests 400000
Fetch example json Successful requests 400000
Fetch example json Failed requests 0
Fetch example json Median time per request 0ms
Fetch example json Average time per request 0ms
Fetch example json Sample standard deviation 0ms
Concurrency Level 4
Time taken for tests 8.5 seconds
Total requests 400000
Successful requests 400000
Failed requests 0
Requests per second 47323.06 [#/sec]
Median time per request 0ms
Average time per request 0ms
Sample standard deviation 0ms
real 0m8.679s
user 0m20.345s
sys 0m8.674s
ApacheBench without keep-alive
time ab -n 400000 -c 4 http://localhost:8080/json
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>.....
Server Software: wizzardo
Server Hostname: localhost
Server Port: 8080
Document Path: /json
Document Length: 27 bytes
Concurrency Level: 4
Time taken for tests: 20.586 seconds
Complete requests: 400000
Failed requests: 0
Total transferred: 61200000 bytes
HTML transferred: 10800000 bytes
Requests per second: 19430.22 [#/sec] (mean)
Time per request: 0.206 [ms] (mean)
Time per request: 0.051 [ms] (mean, across all concurrent requests)
Transfer rate: 2903.15 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 1
Processing: 0 0 2.3 0 716
Waiting: 0 0 2.3 0 715
Total: 0 0 2.3 0 716
Percentage of the requests served within a certain time (ms)
50% 0
66% 0
75% 0
80% 0
90% 0
95% 0
98% 0
99% 0
100% 716 (longest request)
real 0m20.931s
user 0m3.079s
sys 0m16.728s
ApacheBench with keep-alive
time ab -n 400000 -k -c 4 http://localhost:8080/json
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>....
Server Software: wizzardo
Server Hostname: localhost
Server Port: 8080
Document Path: /json
Document Length: 27 bytes
Concurrency Level: 4
Time taken for tests: 3.141 seconds
Complete requests: 400000
Failed requests: 0
Keep-Alive requests: 400000
Total transferred: 70800000 bytes
HTML transferred: 10800000 bytes
Requests per second: 127331.11 [#/sec] (mean)
Time per request: 0.031 [ms] (mean)
Time per request: 0.008 [ms] (mean, across all concurrent requests)
Transfer rate: 22009.38 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 0 0 0.0 0 2
Waiting: 0 0 0.0 0 2
Total: 0 0 0.0 0 2
Percentage of the requests served within a certain time (ms)
50% 0
66% 0
75% 0
80% 0
90% 0
95% 0
98% 0
99% 0
100% 2 (longest request)
real 0m3.436s
user 0m0.930s
sys 0m2.502s
wrk
./wrk -c 32 -t 4 http://localhost:8080/json
Running 10s test @ http://localhost:8080/json
4 threads and 32 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 331.82us 1.21ms 24.83ms 95.11%
Req/Sec 81.34k 6.73k 89.29k 85.75%
3238606 requests in 10.03s, 472.55MB read
Requests/sec: 323021.10
Transfer/sec: 47.13MB
Thanks @wizzardo for all this information! As soon as we merge the async branch and then we merge the keep-alive(https://github.com/fcsonline/drill/commit/e6b51ffbfecf68f9f5265d54c74e1091a2e10599) I think we will be in a really good throughput position.
I have one more idea to increase the throughput, but it goes after those two merges.
Async and keep-alive features have been merged to master. Can you test again?
Hi there! faced with same problem! Software: macOS Catalina/Node.js/Express.js/PostgreSQL Hardware: Mackbook 2015 i7 2.5Ghz/16Gb/SSD
Case 1 (slow)
threads: 500
base: 'http://localhost:5000'
iterations: 10
rampup: 2
plan:
- name: Get post by id
request:
url: /posts/1
Result:
> drill --stats --quiet --benchmark benchmark.yml
Get post by id Successful requests 5000
Get post by id Failed requests 0
Get post by id Median time per request 941ms
Get post by id Average time per request 2657ms
Get post by id Sample standard deviation 4696ms
Concurrency Level 500
Time taken for tests 40.3 seconds
Total requests 5000
Successful requests 5000
Failed requests 0
Requests per second 124.12 [#/sec]
Median time per request 941ms
Average time per request 2657ms
Sample standard deviation 4696ms
Case 2 (blazing fast)
threads: 500
base: 'http://127.0.0.1:5000' # 'http://0.0.0.0:5000' also fine
iterations: 10
rampup: 2
plan:
- name: Get post by id
request:
url: /posts/1
Result:
> drill --stats --quiet --benchmark benchmark.yml
Get post by id Total requests 5000
Get post by id Successful requests 4915
Get post by id Failed requests 85
Get post by id Median time per request 296ms
Get post by id Average time per request 284ms
Get post by id Sample standard deviation 59ms
Concurrency Level 500
Time taken for tests 3.0 seconds
Total requests 5000
Successful requests 4915
Failed requests 85
Requests per second 1692.92 [#/sec]
Median time per request 296ms
Average time per request 284ms
Sample standard deviation 59ms
Difference in base URL :) Something went wrong when Drill tries resolve DNS. In case IP all fine :)
When I try test live server like super.com/api/my-endpoint performance issue unfortunately still exists. Maybe issue also related from OS, don't know.
p.s. Drill has great yaml config API for performance testing, thanks for this project
Are you trying with the last changes in master? They include async and keep alive (avoid multiple dns resolutions) functionalities that should improve those cases. BTW, remove the rampup option to remove whatever systematic latency. Also swap your current values for iterations and threads.
I tried latest master and don't see big difference
time ./target/release/drill --benchmark benchmark.yml --stats -q
Threads 4
Iterations 100000
Rampup 0
Base URL http://localhost:8080
Fetch example json Total requests 400000
Fetch example json Successful requests 400000
Fetch example json Failed requests 0
Fetch example json Median time per request 0ms
Fetch example json Average time per request 0ms
Fetch example json Sample standard deviation 0ms
Concurrency Level 4
Time taken for tests 7.7 seconds
Total requests 400000
Successful requests 400000
Failed requests 0
Requests per second 51814.33 [#/sec]
Median time per request 0ms
Average time per request 0ms
Sample standard deviation 0ms
real 0m7.942s
user 0m20.503s
sys 0m7.539s
but 52k rps is already pretty good, btw how many connections does it open?
Distro: Ubuntu 20.04 LTS (Focal Fossa) System: Kernel: 5.4.0-29-generic x86_64 Machine: Type: Laptop System: Dell product: XPS 15 9560 CPU: Topology: Quad Core model: Intel Core i7-7700HQ bits: 64 type: MT MCP L2 cache: 6144 KiB
example app for test: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Java/wizzardo-http/README.md runs on java 8 u181
threads: 4 concurrency: 32
time ./target/release/drill --benchmark benchmark.yml --stats -q
Threads 4
Iterations 100000
Rampup 0
Base URL http://127.0.0.1:8080
Fetch example json Total requests 3200000
Fetch example json Successful requests 3200000
Fetch example json Failed requests 0
Fetch example json Median time per request 0ms
Fetch example json Average time per request 0ms
Fetch example json Sample standard deviation 0ms
Concurrency Level 32
Time taken for tests 46.1 seconds
Total requests 3200000
Successful requests 3200000
Failed requests 0
Requests per second 69340.26 [#/sec]
Median time per request 0ms
Average time per request 0ms
Sample standard deviation 0ms
real 0m48.178s
user 2m23.859s
sys 0m40.101s
threads: 4 concurrency: 64
time ./target/release/drill --benchmark benchmark.yml --stats -q
Threads 4
Iterations 10000
Rampup 0
Base URL http://127.0.0.1:8080
Fetch example json Total requests 640000
Fetch example json Successful requests 640000
Fetch example json Failed requests 0
Fetch example json Median time per request 1ms
Fetch example json Average time per request 1ms
Fetch example json Sample standard deviation 1ms
Concurrency Level 64
Time taken for tests 9.3 seconds
Total requests 640000
Successful requests 640000
Failed requests 0
Requests per second 68759.54 [#/sec]
Median time per request 1ms
Average time per request 1ms
Sample standard deviation 1ms
real 0m9.707s
user 0m28.853s
sys 0m8.275s
threads: 1 concurrency: 64
time ./target/release/drill --benchmark benchmark.yml --stats -q
Threads 1
Iterations 10000
Rampup 0
Base URL http://127.0.0.1:8080
Fetch example json Total requests 640000
Fetch example json Successful requests 640000
Fetch example json Failed requests 0
Fetch example json Median time per request 3ms
Fetch example json Average time per request 3ms
Fetch example json Sample standard deviation 2ms
Concurrency Level 64
Time taken for tests 28.8 seconds
Total requests 640000
Successful requests 640000
Failed requests 0
Requests per second 22240.46 [#/sec]
Median time per request 3ms
Average time per request 3ms
Sample standard deviation 2ms
real 0m29.159s
user 0m22.280s
sys 0m6.870s