tomaka
Performance compared to Iron
There seems to be a noticeable performance difference when comparing Rouille with Iron. The following are numbers I've pulled from my old Macbook Pro:
Rouille:
extern crate rouille;
use rouille::{Response};
fn main() {
rouille::start_server("localhost:3000", move |request| {
Response::text("Hello, World")
});
}
Using wrk -t 2 -c 400 http://localhost:3000 I get:
Running 10s test @ http://localhost:3000
2 threads and 400 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 32.14ms 12.41ms 103.73ms 60.62%
Req/Sec 5.71k 1.19k 7.57k 62.50%
113689 requests in 10.01s, 16.70MB read
Socket errors: connect 0, read 278, write 0, timeout 0
Requests/sec: 11357.80
Transfer/sec: 1.67MB
Now for Iron:
extern crate iron;
use iron::prelude::*;
use iron::status;
fn main() {
fn hello_world(_: &mut Request) -> IronResult<Response> {
Ok(Response::with((status::Ok, "Hello World!")))
}
let _server = Iron::new(hello_world).http("localhost:3001").unwrap();
println!("On 3001");
}
And wrk output:
Running 10s test @ http://localhost:3001
2 threads and 400 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 778.95us 412.51us 16.91ms 81.08%
Req/Sec 16.84k 4.80k 28.63k 72.00%
335056 requests in 10.02s, 36.43MB read
Socket errors: connect 0, read 358, write 0, timeout 0
Requests/sec: 33454.01
Transfer/sec: 3.64MB
Memory-wise, Rouille starts around 612K and during testing climbs to 31MB. Eventually ending around 18MB.
Iron starts at 1.3MB and during testing climbs to 3.5MB and pretty much stays there.
Latency between the frameworks is also pretty drastic. I'm not well versed in benchmarks but the numbers are greater in difference than I would have imagined.
Any idea what could be causing this?
First of all the differences are likely caused by the fact that rouille doesn't use hyper but tiny-http because I noticed that hyper tends to randomly drop some requests, which is obviously not great. I don't think rouille or iron cause anything more than a bit of noise. Tiny-http was written before hyper but never got any hype from the community is not maintained as well as hyper.
The number of requests per second is easily explained by the fact that hyper uses mio (which uses epoll), which is obviously faster than tiny-http which uses the TcpServer from the standard library.
When hyper was using std::io::TcpServer the difference is performances was lower. I still remember benchmarking hyper at ~85kreq/sec and tiny-http at ~65kreq/sec. Probably that hyper has received some more optimizations since then as well.
Memory-wise, Rouille starts around 612K and during testing climbs to 31MB. Eventually ending around 18MB. Iron starts at 1.3MB and during testing climbs to 3.5MB and pretty much stays there.
Tiny-http/rouille spawns one thread for each client connection, while hyper dispatches connections to a threads pool. The first approach has the drawback that it consumes more memory (since you have more threads), while the second approach has the drawback that you can block the entire server if you have several requests that are long to process (which is what I suspect is happening when I said above that requests were being dropped).
Latency between the frameworks is also pretty drastic. I'm not well versed in benchmarks but the numbers are greater in difference than I would have imagined.
I couldn't explain the latency numbers if the difference was low, but since it's so high for rouille I suspect it's also caused by rouille's design.
More precisely I suspect that rouille processes tons of requests concurrently, which ends up taking a lot of time for each request, while hyper only processes requests four by four, which means that each request takes less time. I suppose wrk starts its timer when the request was actually sent and not when it could have started sending the request.
tomaka, thank you for taking the time to break this down for me. I really appreciate it. A few comments:
- I take it Rouille using hyper is out of the question since it would probably change Rouille's API considerably?
- It sounds like you plan to have rouille be asynchronous at some point and I assume this means making tiny_http utilize mio correct? Is there a timeframe when you think this could happen?
Giving some thought to the situation then, it appears Iron can handle more requests with lower latency than Rouille but is limited by its thread pool (4 x 4). Rouille on the other hand, fires a new thread per request, which provides better concurrency but at the overhead of less requests and more latency. Am I understanding this correctly?
For what it's worth, Rouille is by far my most favorite of the web frameworks I've tried. It just seems to fit my brain better. Working with Iron felt more brittle and a little like pushing a boulder up a hill :)
I take it Rouille using hyper is out of the question since it would probably change Rouille's API considerably?
No, I don't think it would require changes in rouille's API.
At the very beginning rouille was using hyper, and I switched to tiny-http after noticing that hyper would randomly drop some requests.
It sounds like you plan to have rouille be asynchronous at some point and I assume this means making tiny_http utilize mio correct? Is there a timeframe when you think this could happen?
That's very blurry. Tiny-http uses a "pull-based model" (I don't know if this has a real name), as opposed to a "push-based model" that would be required to use mio. I don't think there's a lot of code in tiny-http that could be reused for a port to mio.
Therefore the "better" solution would be to either write an alternative to tiny-http, or to use hyper and fix the bug. Unfortunately hyper wants to switch to tokio/future, and my sixth sense tells me that tokio/future are not the ideal solution either.
This is officially over my head so my opinion on the matter is not really important. Having said that, if community efforts are more guided toward hyper, building off of that would reduce duplicated work. I don't know enough about the tokio/futures route so I can't say one way or another.
It also seems like utilizing hyper may address other issues you have for Rouille (ssl, etc) so it may be a net gain there as well.
I'll trust your reasoning though and assume that your reservations are well founded regarding hyper and are in the best interests of Rouille.
Thanks for taking the time to explain things to me! Feel free to close this issue if the matter is settled.
If it makes any difference, I've opted to build my current project on Rouille. After trying Iron, Nickel, Pencil, and others, Rouille seemed like the most sensible option.
Most of the other frameworks I tried seemed to have little development going on with them. I'm assuming that most existing frameworks that are built off of Hyper are currently in a waiting pattern while things in that area firm up and become less volatile.
Until then, the performance of Rouille is more than adequate for my purposes and it helps me get my project off the ground quicker.
Thanks Tomaka!
Is this issue still of concern that it should stay opened?
The discussion up to this point has value, but if any action is taken based on this issue it could be its own issue that refers this one :)
Anyways great discussion here.
