Kitura-NIO
Kitura-NIO copied to clipboard
Kitura
Poor performance with the web-frameworks benchmark
Kitura with Kitura-NIO seems to performs badly on the web-frameworks benchmarks
Reported over slack. Copied from there:
| Language (Runtime) | Framework (Middleware) | Average | 50th percentile | 90th percentile | 99th percentile | 99.9th percentile | Standard deviation |
|---|---|---|---|---|---|---|---|
| swift (4.2) | perfect (3.1) | 22.11 ms | 21.81 ms | 26.35 ms | 33.46 ms | 56.41 ms | 3607.00 |
| swift (5.0) | vapor (3.3) | 65.25 ms | 27.59 ms | 49.31 ms | 1354.51 ms | 3496.16 ms | 228498.67 |
| swift (5.0) | kitura (2.6) | 35.61 ms | 33.10 ms | 48.56 ms | 74.07 ms | 647.15 ms | 26 433.67 |
| swift (5.0) | kitura (2.7) | 62.88 ms | 61.14 ms | 83.65 ms | 115.44 ms | 533.98 ms | 21219.00 |
| swift (5.0) | kitura-nio (2.7) | 177.09 ms | 65.25 ms | 115.28 ms | 3343.02 ms | 6831.05 ms | 555543.00 |
| Language (Runtime) | Framework (Middleware) | Requests / s | Throughput |
|---|---|---|---|
| swift (4.2) | perfect (3.1) | 44599.00 | 41.93 MB |
| swift (5.0) | vapor (3.3) | 32438.67 | 54.37 MB |
| swift (5.0) | kitura (2.6) | 28 626.67 | 53.09 MB |
| swift (5.0) | kitura (2.7) | 15597.00 | 29.04 MB |
| swift (5.0) | kitura-nio (2.7) | 13049.00 | 24.67 MB |
@pushkarnk this looks like some kind of stall to me. On the 50th percentile (median) kitura/kitura are pretty much the same, but the standard deviation is huge! So either the machine that was running the tests was super busy at some point when running kitura-nio or something stalled/got stuck for a while?
FYI -> docker is in use on above results, then metrics are not very useful (could vary by ~40% )
I'm working on https://github.com/the-benchmarker/web-frameworks/pull/632 to use an isolated droplet and therefore to reduce docker noise
PS : If I can also have sponsored machine from IBM, I'll be happy to run Kitura benchmarks on :heart:
oh docker with networking i/egress networking? Yeah, that'll give you a bunch of variance.
FYI, we can replicate a gap between Kitura-NIO and Kitura-net, and that definitely needs investigation. Previous investigations have shown Kitura-NIO to scale better than Kitura-net, but it's more CPU hungry so slower at lower CPU counts - we need to find out where that CPU is being burnt.
However, @helenmasters and I couldn't replicate the general slowness of Kitura vs. Vapor running the benchmark on a bare-metal Ubuntu 16.04 server, once we removed the Ubuntu version penalty (https://github.com/the-benchmarker/web-frameworks/pull/1346). We limited the Docker containers to a subset of CPUs to emulate the 8-cpu environment, and Vapor and Kitura came out similar in performance (results summarised in https://github.com/IBM-Swift/Kitura/issues/1448).
Perfect did show a lead over the others in our environment, similar to the one reported here, so the puzzle is why our local results for Kitura are substantially better than the ones published above.
