Support llrt bench command

[ahaoboy]

When I want to compare the performance of some functions, it would be convenient if it could support a bench api like vitest

vitest is a bit too complicated. I tried running tinnybench, but it failed because it lacked the Event class.

https://github.com/tinylibs/tinybench https://nodejs.org/api/events.html#class-event

[richarddavison]

Event and EventTarget should be in LLRT as per WinterCG. Here is a basic polyfil for them that can be used as a basis for implementation: https://github.com/mysticatea/event-target-shim/

Simplest implementation would be to extend features here: https://github.com/awslabs/llrt/blob/llrt_modules/src/modules/events/event_target.rs

[ahaoboy]

Event and EventTarget should be in LLRT as per WinterCG. Here is a basic polyfil for them that can be used as a basis for implementation: https://github.com/mysticatea/event-target-shim/

After adding the polyfill, it does work. Interestingly, SIMD on Windows and Ubuntu does not seem to perform as well as the original qjs version.

small json https://github.com/awslabs/llrt/blob/main/package.json big json https://github.com/simdjson/simdjson/blob/master/jsonexamples/twitter.json

// npm install event-target-shim
import { EventTarget, Event } from "event-target-shim";

globalThis.Event = Event
globalThis.EventTarget = EventTarget

import { Bench } from 'tinybench';

const bench = new Bench({ time: 100 });

import mod from './a.json'

async function main() {
  bench
    .add('simd', () => {
      SIMD_JSON.stringify(mod)
    })
    .add('json', async () => {
      JSON.stringify(mod)
    })
  await bench.warmup();
  await bench.run();
  console.log(bench.table());
}

main()

[ {
    Task Name: 'simd',
    ops/sec: '12863',
    Average Time (ns): 77739.62703962662,
    Margin: '±0.93%',
    Samples: 1287
  }, {
    Task Name: 'json',
    ops/sec: '16808',
    Average Time (ns): 59492.50446162962,
    Margin: '±0.84%',
    Samples: 1681
  } ]

[nabetti1720]

This is a bit off topic, but EventTarget seems to work without replacing polyfill.

// index4.js
import { Event } from "event-target-shim";

globalThis.Event = Event

import { Bench } from 'tinybench';

const bench = new Bench({ time: 100 });

import mod from '../package.json';

bench
  .add('json', async () => {
    JSON.stringify(mod)
  })
await bench.warmup();
await bench.run();
console.log(bench.table());

% npm run build4

> build4
> ./node_modules/.bin/esbuild ./src/index4.js --outfile=dist/index4.mjs --platform=node --target=es2023 --format=esm --bundle


  dist/index4.mjs  38.1kb

⚡ Done in 18ms

% npm run llrt4 

> llrt4
> llrt ./dist/index4.mjs

[ {
    Task Name: 'json',
    ops/sec: '151646',
    Average Time (ns): 6594.263105835792,
    Margin: '±0.16%',
    Samples: 15165
  } ]

[richarddavison]

After adding the polyfill, it does work. Interestingly, SIMD on Windows and Ubuntu does not seem to perform as well as the original qjs version.

This is very strange. I got over 200% speedup when implemented SIMD JSON. Can you show me the code?

[ahaoboy]

This is very strange. I got over 200% speedup when implemented SIMD JSON. Can you show me the code?

I found that I used debug mode for testing :(, and in release mode, simd is about 30% faster.

Maybe we can move the json module into llrt_modules as well

+    let json_module = Object::new(ctx.clone())?;
-   // let json_module: Object = globals.get(PredefinedAtom::JSON)?;


+    globals.set("SIMD_JSON", json_module)?;

// import mod from './small.json'
import mod from './big.json'

const str_json = JSON.stringify(mod)
const str_simd = SIMD_JSON.stringify(mod)
const N = 1000

function bench(fn) {
  const st = +Date.now()
  for (let i = 0; i < N; i++) {
    fn()
  }
  const ed = +Date.now()
  return ed - st
}

{
  const simd = bench(() => SIMD_JSON.stringify(mod))
  const json = bench(() => JSON.stringify(mod))

  console.log(
    'simd stringify: ', simd,
    'json stringify: ', json
  )
}

{
  const simd = bench(() => SIMD_JSON.parse(str_simd))
  const json = bench(() => JSON.parse(str_json))

  console.log(
    'simd parse: ', simd,
    'json parse: ', json
  )
}

[nabetti1720]

Hi @ahaoboy. Event class is now supported in #637. This allows tinybench to run with the following simple code, without the need for shim (packages such as tinybench can be loaded directly from node_modules, no bundling required).

// bench.js
import { Bench } from 'tinybench';
import mod from './package.json';

const bench = new Bench({ time: 100 });

bench
  .add('json', async () => {
    JSON.stringify(mod)
  })
await bench.warmup();
await bench.run();

console.log(bench.table());

$ llrt bench.js 
[ {
    Task Name: 'json',
    ops/sec: '199053',
    Average Time (ns): 5023.77333467301,
    Margin: '±0.36%',
    Samples: 19906
  } ]

Also, do you need anything else to close this issue? We may be able to help. :)

[ahaoboy]

Nice~