[WIP] 3D Visualization

[danem]

What changes are proposed in this pull request?

This pull request includes a prototype for 3D visualization capabilities for FiftyOne. This prototype does not yet integrate with the larger FiftyOne frontend, nor does it include features like overlays, modal displays, and more.

How is this patch tested? If it is not, please explain why.

This commit includes some rough benchmarks of decompression and thumbnail generation. The additions of this patch are not yet integrated into the larger FiftyOne app.

Release Notes

Is this a user-facing change that should be mentioned in the release notes?

• [ ] No. You can skip the rest of this section.
• [x] Yes. Give a description of this change to be included in the release notes for FiftyOne users.

Adds initial 3D display prototype.

What areas of FiftyOne does this PR affect?

• [x] App: FiftyOne application changes
• [ ] Build: Build and test infrastructure changes
• [ ] Core: Core fiftyone Python library changes
• [ ] Documentation: FiftyOne documentation changes
• [ ] Other

[danem]

With these commits I can render 3D data within the main FiftyOne app. It is still at a pretty rough state where the following needs to be implemented/improved:

• Proper thumbnail sizing
• Modal state rendering
• Modal interactivity
• Overlays
• Better caching
• Additional compression formats
• ... and more

[danem]

Status Update:

Recording from a 2015 macbook pro. Each point cloud is ~5mb decompressed and ~650kb compressed.

https://user-images.githubusercontent.com/828112/153694763-c98d2eac-aa0c-485b-a7fa-8e3825bd37fd.mov

• Thumbnail rendering is still too slow. Haven't made much effort to optimize yet. I've noticed significant improvements on newer hardware however...
  • This poor performance is probably exaggerated by not immediately rendering when the data is available. When mousing over items, they are almost immediately rendered.
• Modal rendering mode has the basic features. I'm sure there will be some design discussion.
• Mesh and pointcloud colors still aren't handled
• Overlays still aren't implemented.
• Looker controls still aren't implemented

[danem]

Fixed point cloud coloring:

[danem]

Added prototype support for viewing RGBD images:

• Because of how the canvas API works, a javascript PNG parser has to be used to interpret the depth image blobs.
  • This prototype ONLY supports png. Unclear what file types we'd want to support
• Currently implemented without Web Workers. Given that we can use the color data as a thumbnail, mesh generation only occurs when rendering in modal mode. This implementation is likely fast enough for the time being.
• Still need guidance on how to package color and depth data in samples. Code currently assumes a depth_path field along with the standard filepath.

[danem]

I think I've nearly reached the end of what I can do on this front without some additional input. Besides some remaining bugs, and aforementioned performance issues, most of the remaining tasks are questions of design, or reworking other abstractions in the code.

Open Questions:

3D Viewer Functionality

I suggest that the 3D viewer should have at least the following functionality:

• Reset camera button
• Previous and next sample buttons

And maybe the following functionality:

• Additional compression formats
• Camera intrinsic adjustments
• Fly mode
• Automatic point color mapping
• Point rendering size adjustments
• Mesh lighting

A long term nice to have:

• Jump between overlays in a point cloud.

RGBD Functionality

RGBD viewer should have the following functionality:

• Switch between 3D view and 2D depth/color image view
• Support common RGBD formats such as JPEG, PFM, PGM
• Previous and next sample buttons

And maybe the following functionality:

• Remapping depth images to a visible range. (For example)

Overlays

How to incorporate overlays with 3D data deserves a good deal of thought. First, the core Overlay interface needs to be modified to accept a more generic drawing interface. Second, we need to decide what the desired behavior should be when dealing with different types of 3D data. Presumably, the overlays should be visible in the grid view, but given a large point cloud with sparse overlays, this may not be possible to do in a helpful/aesthetically pleasing way. Additionally, RGBD data, users could conceivably have 2D or 3D labels, meaning that overlays will only make sense in certain visualization contexts (although we could just project down any 3D overlay).

Python Side

Some sort of compression mechanism is a must to smoothly navigate large point cloud datasets. During development, I've simply pre-compressed my data sets and tacked on an additional field pointing to the compressed data. This works for the moment, but is hardly ideal from a user experience/correctness point of view.

Furthermore, I'm unsure how to best extend the Sample abstraction to handle data that potentially has multiple files (ie: depth/color, raw/compressed). Both on the python side and the typescript side, there seem to be several flavors of Samples floating around.

Feedback would be greatly appreciated.

[benjaminpkane]

@danem I will give another static pass today or tomorrow and checkout the PR and play with things later this week. You can expect me to review (i.e. checkout the PR and test) once a week. I will try to provide other feedback throughout the week as much as I can.

From an implementation perspective, it might be best to only focus on the expanded view. The thumbnail view will likely need to have an optimized data source (instead of loading the entire point cloud).

[benjaminpkane]

May I ask what your motivations are for this feature? Research? Industry? Side project?

Loving all.the work thus far, but I think it would be beneficial for both parties (you as a contributor and FiftyOne as a project) to align on the goals considering the size of this undertaking. It will be a considerable time investment for both of us (and likely other Voxel51 team members). I wouldn't want things to fizzle out, or be troubled by miscommunication.

[danem]

@danem I will give another static pass today or tomorrow and checkout the PR and play with things later this week. You can expect me to review (i.e. checkout the PR and test) once a week. I will try to provide other feedback throughout the week as much as I can.

The code isn't quite in a "testable" state at the moment given that it relies on some datasets you likely don't have. I'd be happy to provide that to you if you wish. Also, I have taken some time to refactor some components, but there are still some hardcoded variables and hacks floating around that might trip you up.

May I ask what your motivations are for this feature? Research? Industry? Side project?

I'm working on a robotics project (in Industry) and we'd love to use FiftyOne for our CV team. Given that our robots rely on a lot of 3D sensing we thought we could add this missing functionality ourselves.

Given my current progress, I think I could get the minimum functionality implemented in short order if I can resolve some of questions in the in the above comment. I could just make my best guess on some of the functionality and changes required, but it'd be a shame for this to never get merged, so to that end getting your feedback is crucial.

PS. Regarding thumbnails, I think the performance can be significantly better than what is displayed in that video. Like I said, earlier, I haven't spent much time tracking down performance issues, or really understood when calls to renderSelf are made. There is no attempt to cache thumbnails yet, and thumbnails are currently being generated without web workers. So I'm not so sure we'll need a separate source for thumbnail data.

Thanks

[brimoor]

Hey @danem, this is amazing work!! Are you open to a quick call this week so we can discuss your questions and identify the best path forward to make the data model and UI here fit naturally with the rest of the project? Seeing your implementation in action would really help this.

You can reach me at [email protected]

[danem]

Hey @danem, this is amazing work!! Are you open to a quick call this week so we can discuss your questions and identify the best path forward to make the data model and UI here fit naturally with the rest of the project? Seeing your implementation in action would really help this.

Hey @brimoor, sounds great. I sent you an email. I've also joined the slack as danem if you prefer to talk there.

[brimoor]

Hi @danem any chance you can share some example data + code so that others can pick up this work and continue it?

[benjaminpkane]

3d is supported. More to come #3985