Is VM starting time included into benchmark result?

[dolfinus]

Some languages are running inside VMs which can start quite slowly (like JVM or ErlangVM). But I see that only process start and finish times are recorded.

So this startup time is included into the benchmark, even if application itself is not running any calculation during this time. Is that fair?

I mean, it is probably okay in helloworld test, just because it is designed to do nothing and measure time from process start to finish, but for tests with heavy calculations startup time should not be a part of the test.