foxglove
Performance evaluation of new3d vs old3d
Before we start blogging about new3d and considering enabling it by default, we should compare performance with old3d to ensure at minimum that there have been no significant regressions, and ideally to demonstrate improvements.
I would consider this ticket narrower scope, a one time comparison of new3d vs old3d.
For https://github.com/foxglove/studio/issues/956 we should create a framework & infrastructure for evaluating the whole studio app and all panels on an ongoing basis.
@amacneil Do you have something specific in this one-time comparison? Maybe some set of files you want us to try? Part of the beta period is to have our community surface any issues.
Copying thoughts from Slack thread:
Something like playback a file "as fast as possible" and calculate the p50/p90/p99 frame render times would probably be sufficient. Do this for old3d vs new3d, and for several files from our collection, and post the results here.
@amacneil how would playing back data as fast as possible help? Rendering is going to be limited by vsync, so all this would do is skip a lot of data during rendering.
EDIT: @defunctzombie clarified "as fast as possible" in this context means "the time it takes for all extension panels to finish rendering or your monitor vsync, whichever is slower".
One-time comparison of old3d vs new3d using the PR at https://github.com/foxglove/studio/pull/3875
Old3D - Renders the demo scene at 58.16hz with all topics enabled New3D - Renders the demo scene at 59.51hz with the same set of topics enabled
I looked over the new3d code and I don't think these results are quite in line with how old3d behaves. The new3d code calls "done" on the render frame on the react render pass rather than when it actually renders the content of that frame. I think it's worth looking into if these two things can be better connected.
From @jhurliman on 8/23: "By switching to three.js, we can tap into a large body of open-source code that implements different rendering techniques to improve the performance and/or visual fidelity of various renderable types in the 3D panel. So far, what we've done is "switch to three.js" but not the "tap into the large body of different rendering techniques". So the performance of the new 3D panel today is roughly the same as the old 3D panel. Large batches of lines and cubes are a bit faster, and large occupancy grids are currently a bit slower, but everything is in the same ballpark. Any stats or side-by-side comparisons we produce will reflect the current state."
Note for my reference: https://gist.github.com/jtbandes/b1c5f674bea74b2365d512e66f2e557d
Closing this since studio-gym is up and running. We can track work to add more benchmarks in separate tickets.
