node-vs-php-vs-go
node-vs-php-vs-go copied to clipboard
mreschke
Benchmark is misconceiving
I've been using all 3 of these languages a lot over the last few years (PHP5 and PHP7 as for PHP). These benchmarks suffer based on the different implementations the Redis have in the different languages.
Good benchmarks for these kind of broader comparison should compare multiple situations that look a lot more like real-world examples.
My proposal would be to redo this and then actually do:
- base calls with no content
- simulate base API calls (without any external libraries that might affect the language, but only core modules)
- simulate longer-running calls (also based on core modules only)
I'll try to enrich this post later with some real benchmarks, but I have to disappoint you that when you actually do these runs validly, on Go v1.6 (and 1.5 too) Node.JS and PHP will shit their pants by far.
Saying this is PHP's performance is wrong, it's benefitting on an external tool (Nginx/CGIfast/Apache). And added to that saying that caching is enabled is even more cheating. Do you know what happens with Go when you cache native objects and do live recompiles with no down-time? You get nanosecond-results. PHP can't even stay under hundreds of microseconds just by starting up. Even with (semi-implemented) JIT in PHP7 you can't achieve speeds in PHP that would match any compiled implementation.
Finally you're suffocating both Go and NodeJS' performance by using ab. ApacheBench can't handle tens of thousands of requests per second; it will drown in the NodeJS/Go performance of sequential request handling (mostly due filesystem limitations). Use wrk, sniper or vegeta to actually see what the systems are capable of.
I just reread my post and it might come up a bit harsh, but I'm merely trying to portray that what this repo tries to show is misconceiving. PHP7 on it's own is hundreds (sometimes thousands of times) slower than Go. Even behind an optimized server (preferably Nginx) it can't outrun Go. I have a Go API running with a native in-memory caching (without library overhead) and doing requests that easily get responses within a few microseconds (thousands of nanoseconds). And that's even over HTTPS. This is not bragging about my code, but merely showing that PHP is still much slower. Not to forget that if you would use something like Laravel, Symphony, CodeIgniter or ZEND you would have such immense degradation in performance. Also with a library like endless you can make your Go-installation indestructible, guaranteeing no down-time, even on DDoS. We did a DDoS test some weeks ago on a highly optimized Nginx PHP7 installation and it took about 3 seconds and the whole server have a server-load over 150.0 (CentOS) and fully crashed. Go's HTTP package is graceful and won't be daunted by overhead. It will deliver much higher loads of successful responses (Nginx, even on ab returns so many fails over NodeJS and Go). Running endless under Google App Engine with Google's package to auto-upgrade under heavier load can make a Go API written in a few days deliver millions of requests per second. From my experience working in PHP-based companies, for very big customers, you need semi-rocket-scientist that tweak your Nginx/Apache and Linux distro constantly to upkeep all the issues that come by in production. NodeJS and Go are just better and more modern solutions in so many more ways than just performance. Both will save you money, a LOT, trust me :P.
@Allendar Check this out. I made my own test on my development MacBook. Node.js is waaaay faster than PHP https://github.com/borislemke/nodejs_vs_php
@borislemke That's some amazing work! I will keep this thread open and try to see if I can write a similar comparison test with Go 1.7 somewhere this week.
Could you maybe add HTTPS benchmarks to your tests too? It's amazing how different languages deal with TLS, especially over HTTP. I'm really curious what the differences will be on Node.js and PHP. Also; the failure rates on Apache/Nginx (can't see them in your tests) will be pretty high under high loads. I'll show in the Go benchmarks that sending 20k requests will return 20k succes-responses, where PHP under Apache will fail hundreds, sometimes thousands, of those requests.
Just to give you a taste of Go 1.7 otherworldly raw performance over both Node.js and PHP:
package main
import (
"fmt"
"net/http"
)
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
fmt.Print("Hello world")
})
http.ListenAndServe(":8080", nil)
}
$ wrk -d5s -t10 -c5000 http://localhost:8080
Running 5s test @ http://localhost:8080
10 threads and 5000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 6.06ms 0.88ms 157.71ms 75.40%
Req/Sec 5.69k 3.00k 12.14k 58.00%
198228 requests in 5.08s, 21.93MB read
Socket errors: connect 4759, read 84, write 0, timeout 0
Requests/sec: 38996.96
Transfer/sec: 4.31MB
53,525% faster.
Some runs wrk can't even process it (unable to create thread 9: Too many open files).
Update I'm running a 15'inch MacBook Pro 2015 Retina with SSD tho. Maybe that made the difference. Maybe you can test this code on your laptop too.
The formatter even made the performance suffer a bit. Sending output straight to the response writer is even faster:
package main
import "net/http"
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("Hello world"))
})
http.ListenAndServe(":8080", nil)
}
$ wrk -d5s -t10 -c5000 http://localhost:8080
Running 5s test @ http://localhost:8080
10 threads and 5000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 4.99ms 1.05ms 156.05ms 89.01%
Req/Sec 6.05k 4.37k 46.89k 74.56%
241474 requests in 5.10s, 29.48MB read
Socket errors: connect 4759, read 92, write 0, timeout 0
Requests/sec: 47315.60
Transfer/sec: 5.78MB
Was still running Xcode, Safari, Chrome etc and had external screen attached. When running the test with nothing open and hardware unattached wrk won't even run, haha:
$ wrk -d5s -t10 -c5000 http://localhost:8080
unable to create thread 9: Too many open files
$ wrk -d5s -t10 -c5000 http://localhost:8080
unable to create thread 9: Too many open files
$ wrk -d5s -t10 -c5000 http://localhost:8080
unable to create thread 9: Too many open files
$ wrk -d5s -t10 -c5000 http://localhost:8080
unable to create thread 9: Too many open files
Lowering the requests to 2500 kind of gives the same stable results as the previous 5000:
$ wrk -d5s -t10 -c2500 http://localhost:8080
Running 5s test @ http://localhost:8080
10 threads and 2500 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 5.14ms 678.02us 11.98ms 46.16%
Req/Sec 5.23k 2.80k 11.19k 63.54%
238319 requests in 5.10s, 29.09MB read
Socket errors: connect 2259, read 92, write 0, timeout 0
Requests/sec: 46701.91
Transfer/sec: 5.70MB
With the IRIS net/http replacement-library (as the default HTTP library for Go is progressing slower, due to standards and safety etc.) it goes even faster:
package main
import "github.com/kataras/iris"
func main() {
m := iris.New(iris.Configuration{
IsDevelopment: false,
DisableBanner: true,
Gzip: true,
})
m.Get("", func(c *iris.Context) {
c.Write("Hello world")
})
m.Listen(":8080")
}
$ wrk -d5s -t10 -c2500 http://localhost:8080
Running 5s test @ http://localhost:8080
10 threads and 2500 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 3.92ms 536.50us 14.73ms 81.86%
Req/Sec 6.17k 4.28k 13.62k 57.68%
311793 requests in 5.10s, 42.22MB read
Socket errors: connect 2259, read 96, write 0, timeout 0
Requests/sec: 61082.81
Transfer/sec: 8.27MB
Gotta admit on the primes, Node.js seems faster. Not sure if it's my rebuild of the getPrimes call tho or that the C-wrapper in Node.js is faster with maps than Go in this case. Also Go's encode/json library is kind of crappy. Could retry later with ffjson.
Update
Turns out I was looking at the wrong table. Seems to be a bit faster than Node.js (14,290.14 req/s).
package main
import "github.com/kataras/iris"
func main() {
m := iris.New(iris.Configuration{
IsDevelopment: false,
DisableBanner: true,
Gzip: true,
})
m.Get("", func(c *iris.Context) {
c.JSON(200, getPrimes(1000))
})
m.Listen(":8080")
}
func getPrimes(max int) []int {
var i, j int
sieve := map[int]bool{}
primes := []int{}
for i = 2; i <= max; i++ {
if ok, _ := sieve[i]; !ok {
primes = append(primes, i)
for j = i << 1; j <= max; j += i {
sieve[j] = true
}
}
}
return primes
}
$ wrk -d5s -t10 -c2500 http://localhost:8080
Running 5s test @ http://localhost:8080
10 threads and 2500 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 16.30ms 13.93ms 157.14ms 68.41%
Req/Sec 2.12k 1.13k 13.62k 71.23%
73933 requests in 5.10s, 55.14MB read
Socket errors: connect 2259, read 90, write 0, timeout 0
Requests/sec: 14494.08
Transfer/sec: 10.81MB
Here are the final results. Node.js sure is fast on JSON marshaling. Not sure if it's the JavaScript v8 engine of native C wrapper that makes JSON handling so fast. Go can certainly use some work on that; you can obviously see the performance degrading over time in the encoding/json library. I think there's an issue with the read-buffers that give too much overhead.
Test Condition #1
wrk -d5s -t10 -c5000 http://localhost:8080
Simple GET request that returns "Hello World"
/**
* Test results:
* | Rank | Subject | Requests/sec | Data/sec | Avg. Response |
* -------------------------------------------------------------------------------------
* | 1 | Pure Go(Multi Thread) | 60,480.45 | 8.19MB | 3.96ms |
* | 2 | Pure Node.js(Multi Thread) | 20,873.34 | 3.09MB | 11.40ms |
* | 3 | Pure Node.js(Single Thread) | 10,375.05 | 1.53MB | 22.70ms |
* | 2 | Express(JS, Multi Thread) | 3,426.94 | 779.76KB | 68.92ms |
* | 5 | Pure PHP(Multi Thread) | 3,147.95 | 667.17KB | 23.28ms |
* | 6 | Express(JS, Single Thread) | 2,386.69 | 543.06KB | 97.97ms |
* | 7 | Laravel(PHP, Multi Thread) | 26.23 | 101.44KB | 392.64ms |
* -------------------------------------------------------------------------------------
*/
Simple GET request that returns prime numbers between 0 and 1000
/**
* Test results:
* | Rank | Subject | Requests/sec | Data/sec | Avg. Response |
* -------------------------------------------------------------------------------------
* | 1 | Pure Go(Multi Thread) | 14,494.08 | 10.81MB | 16.30ms |
* | 2 | Pure Node.js(Multi Thread) | 14,290.14 | 10.75MB | 16.57ms |
* | 3 | Pure Node.js(Single Thread) | 7,902.13 | 5.95MB | 19.13ms |
* | 4 | Express(JS, Multi Thread) | 3,534.16 | 2.95MB | 66.85ms |
* | 5 | Pure PHP(Multi Thread) | 3,189.39 | 1.43MB | 29.21ms |
* | 6 | Express(JS, Single Thread) | 2,206.12 | 1.84MB | 106.68ms |
* | 7 | Laravel(PHP, Multi Thread) | 44.67 | 60.28KB | 370.60ms |
* -------------------------------------------------------------------------------------
*/
Test Condition #2
wrk -d30s -t10 -c5000 http://localhost:8080
Simple GET request that returns "Hello World"
/**
* Test results:
* | Rank | Subject | Requests/sec | Data/sec | Avg. Response |
* -------------------------------------------------------------------------------------
* | 1 | Pure Go(Multi Thread) | 58,458.82 | 7.92MB | 4.11ms |
* | 2 | Pure Node.js(Multi Thread) | 24,559.01 | 3.63MB | 9.76ms |
* | 3 | Pure Node.js(Single Thread) | 11,629.34 | 1.72MB | 20.63ms |
* | 4 | Express(JS, Multi Thread) | 4,429.92 | 0.98MB | 54.23ms |
* | 5 | Express(JS, Single Thread) | 2,689.57 | 611.98KB | 89.16ms |
* | 6 | Pure PHP(Multi Thread) | 549.22 | 116.91KB | 27.73ms |
* | 7 | Laravel(PHP, Multi Thread) | 35.35 | 226.28KB | 144.51ms |
* -------------------------------------------------------------------------------------
*/
Simple GET request that returns prime numbers between 0 and 1000
/**
* Test results:
* | Rank | Subject | Requests/sec | Data/sec | Avg. Response |
* -------------------------------------------------------------------------------------
* | 1 | Pure Node.js(Multi Thread) | 16,802.80 | 12.64MB | 14.30ms |
* | 2 | Pure Go(Multi Thread) | 14,011.49 | 10.37MB | 17.14ms |
* | 3 | Pure Node.js(Single Thread) | 8,727.14 | 6.57MB | 26.92ms |
* | 4 | Express(JS, Multi Thread) | 4,240.04 | 3.54MB | 56.68ms |
* | 5 | Express(JS, Single Thread) | 2,404.82 | 2.01MB | 99.70ms |
* | 6 | Pure PHP(Multi Thread) | 550.48 | 251.90KB | 27.81ms |
* | 7 | Laravel(PHP, Multi Thread) | 33.48 | 216.63KB | 417.37ms |
* -------------------------------------------------------------------------------------
*/
Hopefully at some point this week I will re-write the entire benchmark. I too need more scenarios anyway. I will build 2 linux VMs on a single machine so no other VM's are running. One VM will run wrk or similar and access the other VM across the local 1gb network (NOT localhost). This removes wrk fighting against the other processes on the server and I want network latency in there too since most of my scenarios are over a 1gb network. I will also test actual DB calls with various databases and libraries. Thanks @Allendar and @borislemke for the info.
I'm looking forward to the tests. On a "raw" perspective it's not of much use tho. Go has surpassed most (native)C/JAVA/Rust benchmarks since 1.6 and in 1.7 they added even faster garbage collection and more efficient assembly resolvers/logic. The only reason Go is knee jerking in both the tests borislemke has made is due to the fact that native net/http and encoding/json is just not that amazing in the core package. There are enough tools like fasthttp and fastjson that can solve this tho. But if you look at the basic (say raw) Hello world calls Go's many times faster than any other language.
This is all web-based performance tho. If you would literally take raw PHP vs raw Go and do some random tests with arrays, maps, heavier calculations and moving data around PHP will drown. PHP is constantly leaking memory and is very at giving it back (I think this is because the whole idea of the language is to handle short requests due to it's web-based nature). A simple loop like this:
$arr = [];
$z = 1;
for ($i = 0; $i < 1e9; $i++) {
$arr[$i] = $z * $i;
}
takes around 123.93 seconds on my MacBook and at the end of the script it has leaked over 2 gigabytes of memory and virtual memory overload. Go only does 3.68 seconds (33,67x faster) over this process and peaks only around 78MB of memory (probably even less due to it's virtual reservation, that is actually not claimed by Go itself yet). PHP has to constantly discover what it's dealing with due to it's dynamic typing (try making $z '1', it's even worse), making it very CPU and memory intensive. It also causes massive memory fragmentation, making the maximum amount of memory that needs to be allocated immense, unnecessarily because most of that memory is just massive empty holes of reserved memory blocks.
@Allendar agreed. I did not see any go tests in @borislemke post but I will test a few packages when I get there.
I am mostly interested in practical usage, not raw performance. Which is why it will be over a 1gb network, with a hard coded array converted to json, a redis query converted to json and a mysql query converted to json etc... I just need reasonably accurate results for a real-world simple server setup that I would actually use. I use these results to determine which API platform I care to build for any particular project. I know my first tests were garbage, so I am looking forward to slapping 2 physical servers on a table and getting some real benchmarks over a LAN.
I know PHP is slower and isn't even in the same ballpark, but we have a million lines of code written in it, so gotta keep it around. We have many APIs written in Laravel's Lumen, even though slow, it's our familiar language. The reason I built this benchmark is to convince myself to build certain critical APIs in something else, less familiar. If anything, it will most likely be nodejs simply becuase we write javascript almost every where now, and its blazing fast. But we'll see once I get it on the LAN with actual DB calls etc...
I understand what you mean. Especially from my older PHP projects (still have some running with external parties too) it's also a good argument for finding employees and having easier to-boot knowledge.
Please try to be cautious (as I noted in my first message in this post), that implementations of Redis, MySQL etc. are very different in all 3 languages. All languages combine a lot of C-wrappers and partially implement fully native code. Go's mysql package (most popular, but not official) is almost native, using only a few C-wrappers. PHP's MySQL implementation is almost all C-wrappers. In a way you're kind of discouraging PHP's outcome in the benchmarks that way too. But again you make a point; if you just want practically comparison (I'm not sure if it can still be called a benchmark then tho) would be nice to see.
Please feel free to post in here when you're working on it (not per se when finished) and maybe we can look at things together and see what makes the comparisons most fair and reasonable.
Ill be sure not to mention its a "raw benchmark" but more of an API server comparison, which is the point. Since its not a raw performance benchmark, if MySQL in PHP is slower, then it's slower. MySQL is usually part of an API in any real-world app I make, so it will be an accurate test as far as my needs are concerned. I'll be sure to set proper expectations of the entire post. Ill keep you updated. My Go is poor so I may need help on a few lines of code.
@mreschke @Allendar Hey guys, I updated my benchmark to include Go + Iris. I separated vanilla Go and Iris to have a more detailed overview. Go wins the Hello world hands down but loses significant performance over Node.js at the prime numbers + json encoding(using intel-go/fastjson).
@borislemke Yeah, like stated, the JSON encoding library isn't that great in Go. I think ffjson is faster than fastjson tho. It fixes a lot of overhead that native encoding/json does in Go. The primes function itself (not returning as JSON), running in a normal benchmark is way faster tho. But like @mreschke said; we're focussing on web-performance here.
To make the tests more believable we might try to make more cases of rendering/loading/stress to actually see the overall performance differences. We already know that "raw" the ranking is Go>Node.js>PHP. To get a real impression on overal use of the 3 languages for web-based performance, we need a broader spectrum (incl. Redis, MySQL, Cookie-handling, Session-handling, TLS, etc.).
I wouldn't do cookies or session as API's should utilize neither. TLS on the other hand...not sure about that as most (mine) API's will utilize SSL termination elsewhere, not in node, nginx, or go itself but in haproxy or some other loadbalancer. The main goal is to simply return a large JSON array, like a large $user object or something. I would test from a hard coded array (no database) then add in a few database libraries like redis and mysql. Either way it will always be json_encode() the same object. Hello world is pointless here.
@borislemke I was also testing vanilla PHP which is 1000x faster than laravel at this simple level. If you wanted an API framework, try Laravel's Lumen instead.
You already know Go's JSON marshaling is not optimal. Just doing more JSON compilation with other jobs in front of it is not much of a use. We already know the performance of the JSON encoding itself.
On the topic of what to test; I thought you wanted to test this as a web-performance indicator, but now you're pointing out it's only for API-usages, which is kind of sub-family of the so-to-say web-family.
In some of my private API's (internal system use) I use BINARY (either over web socket or http) and compress keys into using ENUMs (uint8-based) to identify fields. It's kind of the interpretation of how you would see an API. Even tho in the PHP community JSON seems standard, it's not that standard for the whole world, and like XML, has immense overhead.
Services that software like WhatsApp uses could never serve the 80m+ requests per second if something over the line would need encoding/decoding (besides the line itself (TLS)).
Even in a AngularJS API I have a system where the ENUMs are shared in a TypeScript file by the server to identify keys in the response-JSON with a special decompile function that makes them strings again (if desired for output), thus compressing the bytes that go over the line immensely. This also makes i18n localization so much easier and faster. Browsers are very fast at doing these tasks and you would relieve all the extra stress from the server itself. Just to show you how different you can interpret things Web/API based.
All these points we make is probably also the reason why people always get so hyped(/frustrated) up when seeing compression-benchmarks. There are so many differences between the languages themselves too that often code written in all compared languages are in some way not fully equal and the cases made are not always comparable. I guess you kinda get what I'm trying to aim at ;). To actually show a real comparison, we need to determine the scope what you want to test (HTTP-browser requests, API requests, Websockets, etc.) and what kind of calls and underlying tooling you want to test. After all that you need people that can optimize all the specific benchmarks to work best in their respectable languages. My specialization lies with PHP and Go (of these 3). So in writing a test I would probably discourage Node.js a bit, with some lack of active developing in it. The benchmarks should be free in every language, where each language (and it's optimal coder) should be able to apply all tooling he/she can to make the benchmark perform at it's best in every case. Only then you can probably start to argue about what really makes the difference.
Finally if you're daring, you should never shy of combining the best of multiple languages. I love using Python/Go/Node.js/PHP/C/C++/Objective-C/Swift/etc./etc. in any way they could benefit me in working together. Of course this is kind of a sensitive topic in company environments where only people hired to do one or two of languages could not work with that kind of construct. They're all tools tho; we use what we can to make our software perform at it's best (like elasticsearch, is overhead, but has it's specific tooling speciality, might you need it).
I never planned to take it all the way "web" like this. Perhaps a hello world is all that is needed then. Or perhaps I will add dozens of benchmarks, one of which will be in the context of an actual HTTP JSON API, which was my main intent.
Perhaps @borislemke benchmark is enough. I will still most likely buildout physical machines across 1GB lan for my own tests and my own use case. If you start your own benchmarks for your own personal experiments, I would love to see them.