falcon
falcon copied to clipboard
Benchmarking falcon alongside puma
Hi,
We are benchmarking some ruby
framework here with puma
as appserver
I have recetly discovered falcon
and want to try it (as HTTP/2 is built-in
)
I have compared the performance (disclaimer: on my local
machine : 8 CPU / 16 G).
I have
Framework (Middleware) | Average | 50th percentile | 90th percentile | 99th percentile | 99.9th percentile | Standard deviation | |
---|---|---|---|---|---|---|---|
puma | roda (3.13) | 2.64 ms | 1.97 ms | 5.25 ms | 13.01 ms | 100.45 ms | 2818.67 |
puma | rack-routing (0.0) | 3.39 ms | 2.36 ms | 7.40 ms | 17.56 ms | 64.08 ms | 3567.67 |
puma | flame (4.18) | 5.67 ms | 4.10 ms | 11.77 ms | 29.84 ms | 111.24 ms | 6177.67 |
puma | hanami (1.2) | 6.08 ms | 4.17 ms | 13.95 ms | 29.34 ms | 105.11 ms | 6288.67 |
puma | sinatra (2.0) | 7.37 ms | 5.25 ms | 16.31 ms | 33.66 ms | 101.11 ms | 7176.00 |
puma | rails (5.2) | 26.17 ms | 8.22 ms | 79.53 ms | 162.95 ms | 327.13 ms | 37267.33 |
falcon | roda (3.13) | 40.21 ms | 39.66 ms | 49.50 ms | 88.35 ms | 458.88 ms | 18232.33 |
falcon | rack-routing (0.0) | 47.40 ms | 45.06 ms | 64.39 ms | 90.61 ms | 304.45 ms | 14722.33 |
falcon | flame (4.18) | 61.00 ms | 59.24 ms | 78.09 ms | 138.75 ms | 479.34 ms | 22420.33 |
falcon | sinatra (2.0) | 77.85 ms | 74.01 ms | 103.58 ms | 154.22 ms | 473.21 ms | 25177.33 |
falcon | hanami (1.2) | 115.83 ms | 111.39 ms | 174.22 ms | 247.65 ms | 425.72 ms | 45420.00 |
falcon | rails (5.2) | 243.92 ms | 221.10 ms | 316.00 ms | 1030.75 ms | 1805.75 ms | 164609.00 |
What do you think ?
Regards,
Not looking good. Could you give us some way to replicate the tests?
Sure.
:warning: You need to install crystal
first => https://crystal-lang.org/docs/installation/
- Clone https://github.com/the-benchmarker/web-frameworks
- Run
shards install && shards build --release && bin/neph ruby && bin/benchmarker ruby
:information_source: This is using puma
And you need to
- Clone my fork on the branch use_falcon from https://github.com/waghanza/http-benchmark
- Run
shards install && shards build --release && bin/neph ruby && bin/benchmarker ruby
:information_source: All stuff are running on containers
(wip to change)
@waghanza I just did a bit of local testing, specifically on the sinatra one. I think I may know why you're getting those results.
I hope I am not mistaken here; but I don't believe the setup laid out in the config.ru
is correct, and in my testing with rackup
it defaulted to webbrick. @ioquatix may know how to do this without changing much.
However, to "fix" this, there is the falcon
command (which is installed with the gem) which I used that dramatically changed the performance of the server.
Moreover, it's important to remember that HTTP2 is on by default, and this requires SSL which isn't enforced by the other frameworks; but this can be turned off with a proper --bind
flag:
$ falcon serve -c config.ru --bind http://localhost:9292
Note: if I use rackup
for the puma example in the repo without changing anything, it uses puma under the hood. So, I imagine this is a point of contention for sure, since this isn't the case for falcon.
In my stupidly simple apache bench test:
$ ab -n 5000 -c 100 http://localhost:9292/
Puma Perf
Started with rackup
Server Software:
Server Hostname: localhost
Server Port: 9292
Document Path: /
Document Length: 0 bytes
Concurrency Level: 100
Time taken for tests: 4.209 seconds
Complete requests: 5000
Failed requests: 0
Total transferred: 860000 bytes
HTML transferred: 0 bytes
Requests per second: 1187.87 [#/sec] (mean)
Time per request: 84.185 [ms] (mean)
Time per request: 0.842 [ms] (mean, across all concurrent requests)
Transfer rate: 199.52 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.4 0 5
Processing: 9 83 8.3 82 126
Waiting: 7 81 8.3 81 125
Total: 12 83 8.1 82 126
Percentage of the requests served within a certain time (ms)
50% 82
66% 84
75% 84
80% 85
90% 88
95% 95
98% 109
99% 117
100% 126 (longest request)
Falcon Performance
Started with falcon serve -c config.ru --bind http://localhost:9292
Server Software: falcon/0.18.14
Server Hostname: localhost
Server Port: 9292
Document Path: /
Document Length: 0 bytes
Concurrency Level: 100
Time taken for tests: 1.420 seconds
Complete requests: 5000
Failed requests: 0
Total transferred: 1245000 bytes
HTML transferred: 0 bytes
Requests per second: 3521.11 [#/sec] (mean)
Time per request: 28.400 [ms] (mean)
Time per request: 0.284 [ms] (mean, across all concurrent requests)
Transfer rate: 856.21 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 2 1.6 2 7
Processing: 2 25 22.1 21 184
Waiting: 1 12 15.1 8 182
Total: 2 28 22.4 23 185
Percentage of the requests served within a certain time (ms)
50% 23
66% 28
75% 32
80% 35
90% 44
95% 55
98% 138
99% 147
100% 185 (longest request)
With HTTPs, things are different (since there is literally more being done for each request):
Server Software: falcon/0.18.14
Server Hostname: localhost
Server Port: 9292
SSL/TLS Protocol: TLSv1.2,ECDHE-RSA-CHACHA20-POLY1305,2048,256
TLS Server Name: localhost
Document Path: /
Document Length: 0 bytes
Concurrency Level: 100
Time taken for tests: 38.872 seconds
Complete requests: 5000
Failed requests: 0
Total transferred: 1245000 bytes
HTML transferred: 0 bytes
Requests per second: 128.63 [#/sec] (mean)
Time per request: 777.437 [ms] (mean)
Time per request: 7.774 [ms] (mean, across all concurrent requests)
Transfer rate: 31.28 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 12 527 108.3 543 795
Processing: 18 242 103.3 218 714
Waiting: 10 104 126.0 46 692
Total: 56 769 59.4 761 1190
Percentage of the requests served within a certain time (ms)
50% 761
66% 766
75% 770
80% 772
90% 788
95% 807
98% 974
99% 1136
100% 1190 (longest request)
I just noticed the Dockerfile
CMD
to see that both puma
and falcon
commands are used in the use_falcon
branch.
(This is correct behavior, as far as I know.)
@picatz on which framework ?
@waghanza sinatra
@picatz I don't have that in https://github.com/waghanza/http-benchmark/blob/use_falcon/ruby/sinatra/Dockerfile
@waghanza Right, I was wrong to use the rackup
to test. I apologize for the confusion.
When I use puma
and falcon
~as they are in the Dockerfiles~:
Puma
$ puma -p 3000 -e production
Server Software:
Server Hostname: 0.0.0.0
Server Port: 3000
Document Path: /
Document Length: 0 bytes
Concurrency Level: 100
Time taken for tests: 1.668 seconds
Complete requests: 5000
Failed requests: 0
Total transferred: 860000 bytes
HTML transferred: 0 bytes
Requests per second: 2997.75 [#/sec] (mean)
Time per request: 33.358 [ms] (mean)
Time per request: 0.334 [ms] (mean, across all concurrent requests)
Transfer rate: 503.53 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.4 0 5
Processing: 2 33 2.6 33 41
Waiting: 2 32 2.6 32 40
Total: 7 33 2.3 33 42
Percentage of the requests served within a certain time (ms)
50% 33
66% 33
75% 34
80% 34
90% 35
95% 36
98% 38
99% 39
100% 42 (longest request)
Falcon
$ falcon serve -c config.ru --bind tcp://0.0.0.0:3000
Server Software: falcon/0.18.14
Server Hostname: 0.0.0.0
Server Port: 3000
Document Path: /
Document Length: 0 bytes
Concurrency Level: 100
Time taken for tests: 1.384 seconds
Complete requests: 5000
Failed requests: 0
Total transferred: 1245000 bytes
HTML transferred: 0 bytes
Requests per second: 3613.98 [#/sec] (mean)
Time per request: 27.670 [ms] (mean)
Time per request: 0.277 [ms] (mean, across all concurrent requests)
Transfer rate: 878.79 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 1.2 1 6
Processing: 1 25 23.4 20 186
Waiting: 1 13 17.6 7 184
Total: 2 27 23.7 21 190
Percentage of the requests served within a certain time (ms)
50% 21
66% 26
75% 31
80% 34
90% 46
95% 62
98% 137
99% 145
100% 190 (longest request)
@waghanza I got an error in benchmarker because kore.io
did not have a language version. I had to change benhmarker around line 74 to:
if row.has_key?("language") && row.has_key?("version")
target = Target.new(lang.as_s,
framework.as_s,
"http://#{link.to_s}", row["version"].to_s,
row["language"].to_s)
targets.push(target)
end
Here is the error I was getting:
Unhandled exception: Missing hash key: "language" (KeyError)
from /usr/local/Cellar/crystal/0.26.1_1/src/gc/boehm.cr:71:5 in 'frameworks'
from tools/src/benchmarker.cr:180:1 in '__crystal_main'
from /usr/local/Cellar/crystal/0.26.1_1/src/kernel.cr:386:5 in 'main'
@aemadrid Thanks for reminder, I've fix that on https://github.com/the-benchmarker/web-frameworks/pull/481
Nice work everyone I will take a look when I have time.
I have now (env : 4 CPU x 8 Go)
-
wrk http://0.0.0.0:3000
falcon
===============================
Running 10s test @ http://0.0.0.0:3000
2 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 0.85ms 727.69us 20.33ms 94.77%
Req/Sec 6.10k 2.90k 9.43k 47.00%
121338 requests in 10.00s, 26.61MB read
Requests/sec: 12132.35
Transfer/sec: 2.66MB
puma
===============================
Running 10s test @ http://0.0.0.0:3000
2 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 646.64us 807.83us 21.56ms 94.33%
Req/Sec 9.47k 570.28 10.20k 76.00%
188404 requests in 10.00s, 30.90MB read
Requests/sec: 18837.14
Transfer/sec: 3.09MB
- our benchmark (
wrk
on 3 rules https://github.com/the-benchmarker/web-frameworks/issues/221#issue-328459520)
Framework (Middleware) | Average | 50th percentile | 90th percentile | 99th percentile | 99.9th percentile | Standard deviation | |
---|---|---|---|---|---|---|---|
puma | roda (3.13) | 1.74 ms | 0.19 ms | 4.21 ms | 27.84 ms | 88.72 ms | 5173.33 |
puma | rack-routing (0.0) | 2.12 ms | 0.35 ms | 5.13 ms | 30.27 ms | 103.19 ms | 5662.33 |
puma | flame (4.18) | 3.62 ms | 0.54 ms | 10.13 ms | 39.80 ms | 98.39 ms | 7542.67 |
puma | sinatra (2.0) | 4.12 ms | 1.16 ms | 10.07 ms | 49.56 ms | 120.51 ms | 8978.00 |
puma | rails (5.2) | 18.39 ms | 2.55 ms | 63.96 ms | 118.43 ms | 256.87 ms | 28863.00 |
puma | hanami (1.2) | 5.87 ms | 4.42 ms | 10.59 ms | 29.85 ms | 174.60 ms | 6269.00 |
falcon | roda (3.13) | 58.55 ms | 52.60 ms | 86.29 ms | 136.56 ms | 783.38 ms | 29647.00 |
falcon | rack-routing (0.0) | 65.72 ms | 63.42 ms | 83.80 ms | 133.13 ms | 642.12 ms | 26455.33 |
falcon | sinatra (2.0) | 102.68 ms | 82.64 ms | 151.66 ms | 470.13 ms | 1613.69 ms | 93714.67 |
falcon | flame (4.18) | 96.56 ms | 86.32 ms | 145.91 ms | 277.76 ms | 942.45 ms | 58451.00 |
falcon | hanami (1.2) | 146.52 ms | 134.39 ms | 231.37 ms | 314.18 ms | 757.97 ms | 65566.33 |
falcon | rails (5.2) | 328.06 ms | 320.40 ms | 386.02 ms | 1097.99 ms | 2080.99 ms | 163640.33 |
For me falcon
perform better than puma
, but using docker
is misleading, I will follow our update / change for docker
to instances
There are two aspects of this benchmark which I think needs to be clarified:
-
Falcon is still
< 1.0
and I haven't invested a huge amount of effort into raw performance. -
Falcon is a server built around concurrency. That means I was more interested in this kind of graph.
I think setting up benchmarks like these is awesome and a great way to push development (and understanding). Puma is obviously a lot more mature than Falcon.
One final thing to keep in mind, is that Puma is typically run behind Nginx. Falcon is designed to be run directly connected to the client. Internally Falcon will probably end up with a proxy (need to receive the request and forward to a different process). That being said, the way this affects performance in production is ultimately something worth discussing and testing.
@ioquatix Sure, puma
is more mature. That's why I will no merge to master :stuck_out_tongue_closed_eyes:
In our benchmark, we use raw
puma
(no nginx
) -> keep track of falcon
on our production upgrade (we will run on http://digitalocean.com 8 CPU x 32 Go)
Do you mind benchmarking the latest version? There have been quite a few performance enhancements since last time.
Also at this point I'd be happy for you to merge in falcon support.
HAH! Puma just released v4 which uses nio4r :) You are onto something @ioquatix and I am super excited about where you are going to take falcon now :)
I hope I am not mistaken here; but I don't believe the setup laid out in the
config.ru
is correct, and in my testing withrackup
it defaulted to webbrick. @ioquatix may know how to do this without changing much.
Note: if I use
rackup
for the puma example in the repo without changing anything, it uses puma under the hood. So, I imagine this is a point of contention for sure, since this isn't the case for falcon.
@waghanza Right, I was wrong to use the
rackup
to test. I apologize for the confusion.
You can use this option
-s, --server SERVER serve using SERVER
rackup -s puma
rackup -s falcon
Maybe a good idea to simplify the project behind, but as I understand using bundle exec rackup
is just a wrapper around bundle exec puma
or bundle exec falcon
Actually rackup cannot take full advantage of the server design, so I advise against using it for proper benchmarks.