depthai-unity
depthai-unity copied to clipboard
luxonis
Roadmap
Why?
Sharing the roadmap with everyone is always good for transparency and best way to drive the development in order to bring the most value in each step.
What?
All feedback is more than welcome ! And we'd love to know more about you and get some insights:
- Main platform are you using? (MacOS, Win, Linux)
- Could you explain bit more about your use case? What are you looking to build with the Unity plugin?
- What part of the Unity plugin you find most interesting? (OAK for Creators, OAK for Developers, OAK for CV/AI)
- Do you have experience with OAK API/SDK? What platform are you using the most? (C/C++ , Python)
Features and Roadmap:
v1.0.0 Initial Release to AssetStore (free):
- No code approach. Just drag and drop some prefabs. Bullet-proof.
- Access to OAK device streams:
- RGB and depth (point cloud viz currently ongoing dev).
- About point cloud: Add support for external libraries: PCL, ... and VFX integration.
- Access to IMU if available and retrieve system information.
- Record and Replay capability.
- RGB and depth (point cloud viz currently ongoing dev).
- OAK Device Manager and multiple OAK devices support.
- OAK For Unity Creators: High-Level API - Unlock “Myriad” applications with predefined and ready-to-use pretrained models (with depth support if you run OAK-D family):
- Face Detector, 3D Head Pose Estimation, Face emotion, Body Pose (MoveNet), Hand Tracking and Gesture recognition (Blaze Hands), Object detector (Tiny Yolo v3/4/5 and MobileNet SSD). DeepLabv3 segmentation. Background removal, Depth (only OAK-D family) - point cloud visualization
- Multiple camera demo.
- Example how to integrate your own pipeline
- Integration with some paid assets (demos)
- OAK For CV/AI/Robotics: Unity Virtual OAK Camera
Version 1.0 is just the top of the iceberg. Foundations to build applications on top. Beta testing along the path to get v1.0 ready for assetstore.
Some topics for next versions (big blocks without much detail)
- More complex demos on top of pretrained models (see list below)
- Send unity virtual camera images to OAK pipeline.
Do you miss some models / use cases / integration with some specific paid asset? Please let us know.
- Android support. Integration with AR.
- End-to-end workflows for synthetic dataset generation, training and deploy.
- Integration with Unity Robotics Hub (ROS) / SystemGraph / SensorSDK. Publish to ROS.
- Integration with Unity Simulation and Unity Simulation Pro.
- Intergration with Unity ML-Agents.
- Define custom pipelines inside Unity (visual tool).
- Implement partial C# wrapper
How?
We're opening beta testing on the repo. The plan is start delivering all the features gradually following initial order and always covering full feature (unity plugin lib + unity project/demos) and for all platforms (Win, Mac and Linux)
Following features (v1.0) are ready:
- Device Manager
- Demo menu scene
- Demos:
- Device streams (rgb, depth), Point cloud visualization using VFX
- Object detector demo
- Face detector demo
- Body pose demo
- Head pose demo
- Face emotion demo
- Hand tracking demo
Please tell us about your project !
More resources: unity forums: https://forum.unity.com/threads/oak-for-unity-spatial-ai-meets-the-power-of-unity.1205764/ youtube unity plugin playlist: https://youtu.be/CSFOZLBV2RA?list=PLFzqMMJPSNSbsHp7QeJpOHrZu_1BAdDms
I use Windows. I'm looking forward for the Unity plugin to create games using the human pose estimation and I don't have any experience with the OAK api/sdk 😅
My idea is a game where you can move (a lot). There will be objects falling from above and you can use any part of your body to repel them. The objective is to avoid objects from touching the ground
Hi @MiguelPereyraC Thanks a lot for the feedback. Definetly human pose is going to be great and fun use case with Unity :) I have more feedback about interest on body pose so it's prioritized. About the plugin is coming with precompiled libraries so you don't need to deal with C/C++ for pretrained models.
About body pose here is sneak peak: https://youtu.be/cCeF87glJh4
predefined/pretrained models are just prefabs you can drag&drop and build on top. In this example I was doing the squid game (red light/green light)
In the meantime here some more resources: discord: https://discord.gg/4hGT3AFPMZ unity forums: https://forum.unity.com/threads/oak-for-unity-spatial-ai-meets-the-power-of-unity.1205764/ youtube unity plugin playlist: https://youtu.be/CSFOZLBV2RA?list=PLFzqMMJPSNSbsHp7QeJpOHrZu_1BAdDms
Any C# Examples?
Hi @acidhax ! Could you share bit more detail about your use case and platform using?
Hi @acidhax ! Could you share bit more detail about your use case and platform using?
Prototyping AR experiences in Unity
Hi @acidhax ! Could you share bit more detail about your use case and platform using?
Prototyping AR experiences in Unity
About C# examples, just want to share that Unity plugin library is based on depthai-core but it's not a C# wrapper, if you're looking for that. In first stage depthai pipelines are developed in C/C++ and create interface for Unity C#. More in long term the idea is to create the pipelines inside Unity.
@acidhax @gespona I think we would actually be better off creating C# wrapper (at least partial) along the Unity efforts, to also take better care with exceptions between languages, etc..
If anyone has any good recommendations for such tool/library let us know. We currently use pybind11 for Python wrapper and community driven Java bindings use JavaCPP.
@acidhax @gespona I think we would actually be better off creating C# wrapper (at least partial) along the Unity efforts, to also take better care with exceptions between languages, etc..
If anyone has any good recommendations for such tool/library let us know. We currently use pybind11 for Python wrapper and community driven Java bindings use JavaCPP.
Yes I agree. Exceptions management currently is bit pain. Partial C# wrapper sounds good approach and probably similar approach for this mid-long term goal of defining pipelines inside Unity with visual node tool. Did PoC in the past and basically was exposing part of the core C++ objects for pipeline creation.
As Martin said if there is any recommendation, feel free to share here
I just updated the roadmap adding Partial C# wrapper for better integration on Unity side.
Will do small PoC with these two options:
https://github.com/mono/CppSharp https://github.com/swig/swig
Please feel free to share other options or any experience with them :)
Is the Squid Game demo on the repo? I really want to try it out! 😁
Not yet but will be ;) Will include all the demo scenes you can find today on weekly video updates (+prebuild binaries). Some paid assets not going to be availabe but replaced with free ones and instructions on documentation how to integrate paid assets if you want to reproduce exactly the demo scene.
In order to make more clear the state of the repo, I will update How section with progress. Also you will find on README in "beta" branch. I will merge first with Windows10 support and then for rest of platforms (Mac/Linux)
Quick update here: "beta" branch updated with basic streams and point cloud vfx for Windows. Working on macos/linux support. Think almost all the feedback received is on Windows platform, depends on next feedbacks moving faster with pretrained models on Windows ;)
I'm an augmented reality developer for Windows Unity. I can not wait for the Oak for Unity plugin because of the possibilities these cameras have with range in general. Putting two up on opposite corners of the room would give me so many possibilities for controlling the AR environment with greater accuracy. The problem with most AR headsets is that they only track your hands when they see your hands. So setting up these cameras to interact with Unity would mean that my hands, body, pose, posture, etc would ALWAYS be tracked and available to use as peripherals
I'm an augmented reality developer for Windows Unity. I can not wait for the Oak for Unity plugin because of the possibilities these cameras have with range in general. Putting two up on opposite corners of the room would give me so many possibilities for controlling the AR environment with greater accuracy. The problem with most AR headsets is that they only track your hands when they see your hands. So setting up these cameras to interact with Unity would mean that my hands, body, pose, posture, etc would ALWAYS be tracked and available to use as peripherals
Hi ! @treyDavisMCC I can't agree more with you ! ;) OAK + Unity = endless possibilites in terms of HCI / AR / Interactive ... you can check few experiments I've doing in YT playlist. Also in the past I was merging two OAK cameras in same unity 3D space.
If you're on windows, please check 'beta' branch .. right now you can get small sneak peek of what's coming soon ! ;)
@gespona I was actually watching your experiments. You're a much more experienced coder when it comes to machine learning. All I can think of are applications of other people's machine learning. You're out here creating realistic water flow with it. I really need to take a class on this or start working entry level at a company for this, because this is so over my head I do not think I could learn it by myself.
@gespona I was actually watching your experiments. You're a much more experienced coder when it comes to machine learning. All I can think of are applications of other people's machine learning. You're out here creating realistic water flow with it. I really need to take a class on this or start working entry level at a company for this, because this is so over my head I do not think I could learn it by myself.
Thank you for your kind words ;) Btw just want to clarify or bring some light here if you're talking about the hand tracking + depth + fluid demo ... Credits for fluid simulation are for Zibra AI Liquids ;) . This is another nice thing of Unity as you can use great assets (free or paid). So for my experiments you'll see often I'm combining with other assets to give a meaning to the demo (p.eg: airplane controller with Overcloud, nautilus demo with CrestOcean, photobooth demo, ...)
Anyway I will add all demos to the plugin, and instructions to download extra assets if need it.
I plan also to create lot of tutorials to help everyone to get the maximum profit of the plugin and OAK devices potential, much further of my demos (it's just top of iceberg). And feel free to contact me on Discord if you need specific help.
@gespona I am struggling to set up the beta on my unity. I have tried cloning the git into a folder to open in unity, and I've tried cloning it into an already made Unity project. Neither worked when for the purposes of step two in the OAK for Unity section of your md
@gespona I am struggling to set up the beta on my unity. I have tried cloning the git into a folder to open in unity, and I've tried cloning it into an already made Unity project. Neither worked when for the purposes of step two in the OAK for Unity section of your md
Hi ! I just tested again, everything looks good but opened this issue with following steps for you: https://github.com/luxonis/depthai-unity/issues/5
I'm still working on documentation (pretty behind on this). After solving your issue I will update docs with more detailed info and screenshots about the steps.
Successfully tested Beta Branch with Oak-D Lite on Windows 11 Enterprise v21H2 running on an XPS 13 9300 laptop.
@gespona We are going support OAK-D cameras in our rehabilitation system. We plan to use OAK-D camera for professional (in facility) use, but we are thinking also about home therapy. I wonder whether your plugin (especially body tracking) is planned to work with normal 2D webcams too?
@brontesprocessing This is a really interesting application of the technology and I would definitely be interested in working with you on it potentially via the Djinn project I've got rolling at the moment. I've checked out your website and I can confirm from my own experience of the Oak-D sensors that they would fit your use case perfectly. And the sensor is cheap enough, particularly the entry level sensor, to make home analysis practical and economical. Its something I'd be interested in discussing if you're still looking for help.
Hi @brontesprocessing, First thank you for sharing your project. The scope of the plugin is oak devices, that said when release body tracking (hope very soon) I can give you support to make basic example (2D) work with standard webcam. Please join discord and let's keep in touch there ;)
Hi everyone ! I just updated the repo with some improvements on point cloud. Right now I'm slowing down a bit on the planned roadmap as the second part - synthetic datasets and simulation - got some priority, so hope soon I will start give back some of this development to the repo. That said we keep the same priority list here so body pose coming soon !
Btw, we decided to promote beta branch to main, so from now on just check main branch.
Any update on when the Unity plugin will have the hand-tracking ability?
After reworked a bit the preview modes, now I'm working on head pose and hand tracking , revisiting initial implementation you probably saw in the video (hand track+liquids) . Hand track is by far the heaviest model to run on camera, specially to support two hands, so basically I'm focus on performance now. Could you share bit more about your application / use case?
The immediate use case is to use the camera to allow people to interact with a digital billboard. The board will contain various types of information and we'd like to allow people to use gestures such a grab, swipe, pinch, etc. to move and manipulate the data.
Hi,
First of all great project! Really. I made a post on the Luxonis forum asking about the status of DepthAI for Unity and now I came here to see this is still in works and that's awesome. I'm just going to copy paste my post and formulate this as a question here:
I've been using DepthAI as a python package together with my OAK-D camera for personal projects and I can say I enjoy the experience. i would like to extend those projects and such am wondering if the Unity roadmap is still on the radar for luxonis? The initial forecast seemed very exciting:
v1.0.0 Initial Release to AssetStore (free):
No code approach. Just drag and drop some prefabs. Bullet-proof.
Access to OAK device streams:
RGB and depth (point cloud viz currently ongoing dev).
About point cloud: Add support for external libraries: PCL, ... and VFX integration.
Access to IMU if available and retrieve system information.
Record and Replay capability.
OAK Device Manager and multiple OAK devices support.
OAK For Unity Creators: High-Level API - Unlock “Myriad” applications with predefined and ready-to-use pretrained models (with depth support if you run OAK-D family):
Face Detector, 3D Head Pose Estimation, Face emotion, Body Pose (MoveNet), Hand Tracking and Gesture recognition (Blaze Hands), Object detector (Tiny Yolo v3/4/5 and MobileNet SSD). DeepLabv3 segmentation. Background removal, Depth (only OAK-D family) - point cloud visualization
Multiple camera demo.
Example how to integrate your own pipeline
Integration with some paid assets (demos)
OAK For CV/AI/Robotics: Unity Virtual OAK Camera
Do you think we can expect a release to the asset store in 2024, or is this not of a high priority right now?
Hi,
First of all great project! Really. I made a post on the Luxonis forum asking about the status of DepthAI for Unity and now I came here to see this is still in works and that's awesome. I'm just going to copy paste my post and formulate this as a question here:
I've been using DepthAI as a python package together with my OAK-D camera for personal projects and I can say I enjoy the experience. i would like to extend those projects and such am wondering if the Unity roadmap is still on the radar for luxonis? The initial forecast seemed very exciting:
v1.0.0 Initial Release to AssetStore (free): No code approach. Just drag and drop some prefabs. Bullet-proof. .... Do you think we can expect a release to the asset store in 2024, or is this not of a high priority right now?
Hi @AndreiMoraru123 , if you clone this repo you'll find unity project with almost all the roadmap implemented. Right now working on the last examples: hand tracking and head pose. Regarding AssetStore yes, I hope to publish after complete the examples (before EoY) .. that said the main content is here in the repo and under releases you have also some examples as .unitypackages