studio icon indicating copy to clipboard operation
studio copied to clipboard

Benchmark generator script

Open jhurliman opened this issue 1 year ago • 1 comments

User-Facing Changes

  • None

Description

This adds a benchmark generator script. See the included README for execution instructions. The current generator only produces four topics: /tf, and a list of cubes represented in three different ways. The length of each cube list is tuned to stress my MacBook Pro M1.

jhurliman avatar Aug 02 '22 03:08 jhurliman

Few thoughts:

  1. It would be nice for the script to have some comments on what it is producing - or something documenting that.
  2. Personally I find it hard to read the python notebook style rather than a python script I can run that produces a file tho this is largely preference
  3. Why did we stop at cubes - we have more renderable types and should see how those work as well.
  4. There should be benchmark layouts that correspond to these files - it should be easy for someone to launch the benchmark serve, select a layout, and load the benchmark file.

Do you have numbers produced by running this with benchmark in a layout? Same for other types of renderables.

defunctzombie avatar Aug 03 '22 04:08 defunctzombie

@defunctzombie feel free to merge this while I'm out as a starting point.

jhurliman avatar Aug 28 '22 14:08 jhurliman

I'll take a look. I don't think I want to merge this as-is (per my comments above). I think there's something more natural to create here for our typescript workflow.

defunctzombie avatar Aug 29 '22 02:08 defunctzombie

We've taken a different tactic with our benchmarks. The data for benchmarks is generated synthetically via custom players in the benchmark folder. Closing this in favor of the approach we have already deployed.

defunctzombie avatar Nov 18 '22 05:11 defunctzombie