FrameworkBenchmarks
FrameworkBenchmarks copied to clipboard
TechEmpower
New classification for implementation types
I would like to suggest a new way to classify the implementations of this benchmark, primarily to exclude those implementations which are just simple wrappers over existing libraries implemented in other languages as I believe they are not in the spirit of these benchmarks.
Implementations which come under this category include at least pico.v and h2o.cr, but there may be more. I think it would be ideal if these implementations were filtered out from the default results page, but if not then at the very least there should be a way to filter them out explicitly if the person viewing the results chooses to do so.
One way to determine whether implementations belong to this category would be to check whether they require libraries (written in other programming languages) that are not ordinarily needed by other software written in that language. As an example, code written in a language like Nim where libc is always a dependency is fine, whereas code written in V depending on pico is not since that is not necessary for all code written in V.
I think doing this will allow the benchmarks to truly show the power of the languages that they are implemented in, rather than who can wrap the fastest library written in C.
I think doing this will allow the benchmarks to truly show the power of the languages that they are implemented in, rather than who can wrap the fastest library written in C.
What is the value here? If I am a developer looking to start a webapp in V, do I care if the framework I choose uses C bindings under the hood? I tend to think not.
In fact, I would argue the opposite - if my Java platform uses C under the hood but I am writing Java and am unaware (but are reaping improved performance vs other Java frameworks), then I am happier as a dev.
I don't disagree with the request being made here - you'd like to know whether a framework is "pure" for a given language. I have no opinion about this, and it would be trivial to add to benchmark_config.json.
I don't disagree with the request being made here - you'd like to know whether a framework is "pure" for a given language. I have no opinion about this, and it would be trivial to add to
benchmark_config.json.
We already have a mechanism for tags now in the benchmark_config.json, so I'm fine with thinking of a tag for this and adding it there, or some other way to make it easier to filter, but I'm definitely against hiding these implementations from the default results.
What is the value here? If I am a developer looking to start a webapp in V, do I care if the framework I choose uses C bindings under the hood? I tend to think not.
That depends on what you're evaluating. Often those that see these benchmarks use them to get an insight into how fast a particular programming language is, and in that case such implementations are very misleading.
We already have a mechanism for tags now in the benchmark_config.json, so I'm fine with thinking of a tag for this and adding it there, or some other way to make it easier to filter,
Awesome. It seems like "pure" would be a good initial idea, but we should probably make it more specific. Any ideas?
I don't really like "pure" either. Maybe there's some kind of secondary-language tag or something. I'm not sure we have the right answer here yet.
How about enhance existing language tag? Like if framework consist of 80% C code and 20% python code then tag show c / py, in contrast case it will be py / c (for example). If framework don't use any ffi and fully written in one language it show tag as usual without slash separation.
Hi es4x author here, in my case it looks like es4x would be excluded because it's a js framework but the VM of choice is GraalVM. I think this label should be carefully designed because in the case of JavaScript, there isn't an single runtime. Yes node is by far the most popular, but there's chakra, there's graalvm and many others.
Also the design idea behind of es4x was to reuse as much existing code as possible so you will find netty and vertx in it, but it's not a simple polyglot binding but a different way to write applications.
Just my 2cts ☺️
