performance version of test262

[leobalter]

trafficstars

Following up @mikeal's tweet, I'm here to say a initial hello to this WG. I'm sorry if I'm using the wrong channel. I'm a TC39 member, representing the jQuery Foundation (that's pretty recent), and a maintainer of test262.

We can have performance tests on test262. TC39 just created another test262 sibling repo for parser tests (see https://github.com/tc39/test262-parser-tests and https://github.com/tc39/test262/pull/559) and I can support creating another one for performance/benchmark tests.

What we need is people with available time to maintain and feed this new project. Plus, we can merge or reuse the work from this WG, as you want, but I'm certain that a sort of questions will be asked and we can anticipate this here: can we have these tests available for any runtime or any different implementation? If that's not possible, can we have tests for node without a specific runtime (e.g. node-charka)?

Count on me to support anything on this process.

[gareth-ellis]

This sounds like the sort of thing we should look into, certainly.

I've had a quick look through some of the tests (i see there's almost 17000!) , and some we should be to turn into performance benchmarks.

In an ideal world, we'd look at using the files like they are, however with the asserts in the files as well i wonder if we'd risk performance testing that, rather than the actual code.

I'll wait for others to chime in here too, however how I'd probably look at doing this is using a harness that can repeat the tests multiple times and we measure the number of times each particular function can be repeated a second. I have a harness already that does this, I'm planning on getting it and some of the benchmarks I have committed to the workgroup soon - just requires a little more work until i'm happy.

to start with, a shortlist of a few tests - say 10? would be good, and we can try and get something working with those first.

If you or someone in your team want to suggest some good candidates (and indeed if there's some that you know there's been a regression in).

[mhdawson]

I think we need to understand why a particular test makes sense to benchmark. We can't benchmark everything and even if we could it would be hard to understand/action the data.

What' I'd like is that suggested tests be put in the context of:

1. helping to validate performance on one of the uses cases here: https://github.com/nodejs/benchmarking/blob/master/docs/use_cases.md
2. helping to validate performance of one of the key runtime attributes https://github.com/nodejs/benchmarking/pull/42

@leobalter a few questions

1. Do you think its possible to get what you have in mind generating 1 or a small set of numbers that could be tracked ?
2. Can you put how benchmarking the test262 fix in the context of the use cases/key runtime use cases

[gareth-ellis]

@leobalter any updates on this?

[leobalter]

I'm sorry for being absent.

We have plans to revive the test262 website and reorganize test run by features subsets and fetch data from automated test runs. That can evolve to time sensitive runs and probably places where we can seek for opportunities to run benchmark tests. This is only a fragment of features that would be interesting to connect test262 to benchmark runs.

I don't have more answers for now but the will to contribute. I'll try to (I should) get more involved with the NodeJS ecossystem.

[mhdawson]

I talked to Boaz Sender from Bocoup at NodeInterative EU who is working on test262 as well. It sounds like tests262 was not written to test performance. We might just run the full test suite and compare times but not sure how useful that would be,

[mhdawson]

@leobalter wondering what the next step on this would be.

[mhdawson]

@leobalter is this still something you are hoping to move forward ?