arcore-android-sdk
arcore-android-sdk copied to clipboard
google-ar
Feature Request: Dense Pointcloud from depth sensors or stereo
I know there were already several requests related to the possible support of Tango Devices with ARCore. My question is based on the fact, that we want to use the already available depth sensors to capture dense depth data with our devices. So is there any plan (short, mid or long term) that the ARCore will use the hardware capabilities of Tango Devices?
Is it possible that ArFrame_acquirePointCloud() not only can deliver depth data about features, but also gives us the pointcloud we already got via tango, when hardware is there?
This is definitely something we could do, but currently ARCore doesn't even run on any devices with depth cameras so it's a bit early to even think about.
Copied from Google Tango homepage:
In addition to working with our OEM partners on new devices, Google is also working closely with Asus, as one of our early Tango partners, to ensure that ZenFone AR will work with ARCore. We'll be back with more information soon on how existing and new ZenFone AR users will get access to ARCore apps.
It seems that at one point ARCore will be running on ZenFone AR. It would be a great feature for the existing hardware.
Yeah i already read that, but to clear things up. This feature request is not about "running ARCore on Asus", that's what i expect. It is more about getting dense pointcloud data from devices which have a hardware component like a Time of Flight Sensor, so ARCore would offer us the best of all development possibilities.
I see the ARCore for the zenfone is eminent. Is this going to have point cloud support? Will the release deprecate the point cloud support that it currently has? I use matterport scenes all the time, I want that to still work but I would also like the new features of ARCore.
How is this going to work?
Sorry, not yet. Point clouds on the ZenFone are derived from visual features, just like on other ARCore phones.
I hope it doesn't override our tango core on the framework, til the support for dense depth data from related sensors, there should be no one be forced to use ARCore. Especially a lot of developers need those data, for exsiting and upcoming applications.
Any update on this with regards to Snapdragon 845 / Galaxy S9 compatibility being unveiled "within weeks"?
No, but certainly lacking clarity.
I meant that one of the features Qualcomm is promoting for the 845 is accurate and dense point clouds based on dual cameras as opposed to IR (through the Spectre 280 ISP). As the S9 has all of the required hardware, and in theory the 845 has software support for that, it seems like this should be a straightforward feature to expose through ARCore.
If anyone hasn't seen the demo video, it is pretty sweet. https://www.youtube.com/watch?v=16vz3_6-tbM
Ah, sorry, didn't read that clearly. Makes sense now.
No, no update on dense depth from stereo. Updated FR description to include.
@inio Any update on this? Asus Zenfone AR is already supported few months.
This app is waiting for it: https://play.google.com/store/apps/details?id=com.lvonasek.arcore3dscanner
How is this progressed?
I think almost all device which has ARCore available contain stereo camera. So the point cloud should be better handled
In this video https://youtu.be/7ZSm95naghw?t=127 is a 3d scanner using depth sensor on non Tango device. Is it based on ARCore? Or shall we migrate to another technology?
In this video https://youtu.be/7ZSm95naghw?t=127 is a 3d scanner using depth sensor on non Tango device. Is it based on ARCore? Or shall we migrate to another technology?
Hi, i also had an eye out on the new oppo. As far as i know this is not based on ARCore, it is even not a open development kit available to use this depth sensor for new apps. I still hope ARCore will offer tango-like functions for phones with the right hardware components, as depth sensors get more common on android devices.
It is written in #603 that they are not going to implement it in the near future. About Oppo I watched the same presentation in more languages. Here was at least some informations shared: https://youtu.be/-Vz5US3vt5E?t=3274
It is written in #603 that they are not going to implement it in the near future. About Oppo I watched the same presentation in more languages. Here was at least some informations shared: https://youtu.be/-Vz5US3vt5E?t=3274
Thx for the link informative video, the question is what does google understands in 'near term', that could be months we have to wait or a year. My hope is that as hardware enabled devices will raise again and the demand of our developer community using that sensors, will force them to give this topic a higher priority.
They already have the knowledge integrating the sensors like they did with tango, maybe it was not perfect but it was working. So why not bringing together the best of both worlds.
The demand from the developers is quite big. Everytime I talk with someone about ARCore I hear words like disappointment, useless, etc... Tango was working well in bussiness area, thats why I tried this request: #638
@lvonasek It not really big at all. Just a computer vision with stereo camera is enough for everything we all need. It was just a stupid paradox decision of ARCore team which I don't know who have think about it
The point is ARCore always limit device it could be run on by no reason. Almost all of it's device has stereo camera like it required to yet they don't utilize stereo camera while it should. ARCore could be run on normal phone with single camera but they just said the quality of those phone is not acceptable for them. However the quality of ARCore that run on my phone right now even the measure app from google is not acceptably accurate or usable
@Thaina Difference between structure from motion with single (current implementation) and pasive stereo camera is that stereo has known transformation from the first photo to the second photo. Then you can reach much better point cloud (accurate). But there would be the same problems like with current implementations - white walls wont be for the device visible. Also occlusion or collision with real world would not work better. Thats why I am saying that demand on depth sensor is quite big - because other systems are just compromises, not full solutions. Of course stereo-camera solution would be better but Google cannot make Google's Pixel rivals better than own device. Maybe Google will solve it later using AI but I am not aware of any ready-to-production system for it.
And about limiting device support - there is really good reason for it. Computer vision needs camera calibration. And Google needs IMU calibration. Google has a lot of work with every supported device (calibration all possible device variants, device firmware fixing, firmware update, whitelisting the device on Google Play). I would say we need patience, Google has enough work to do.
https://www.androidauthority.com/samsung-galaxy-s10-tof-sensor-927054/ "A rear-facing TOF sensor could provide a more accurate augmented reality experience than ARCore"
Would not be better for Google to bring depth sensor standard instead of letting Oppo, Samsung, Huawei and others create own standards? I can imagine that without bringing depth sensor support into ARCore there would be in the future more SDKs and you wont be able to use it together with ARCore because of the locked camera.
Yeah with more TOF equipped phones coming out, this will be a must have. AR without dense depth is just half baked imo. I can't make good use of it. I am forced to use the zed sdk with the zed stereo camera for now to get depth and occlusion. Not ideal for me either, but all i can find for now.
More and more tof camera phones will be coming. AR will only make sense when it takes advantage of that in my opinion. I await that day, and when that day comes I will get excited about AR on phones.
White wall is known that we can't achieve but depth from stereo can solve many more things even without true depth sensor
You can't expect manufacturer to put depth sensor just for AR. It not practical usage for normal customer. But stereo camera is, it would be used in photography and that's all we need
You are underestimate the depth imaging. Just depth image from stereo camera would be exponentially useful. Just the ability to not need to move camera around to start AR session is precious. And it can make a guess on even white wall if we could make a depth on edge between wall and floor. There was so many more usage if we could put reliable depth image in AR
@mpottinger - Huawei already confirmed that ToF support is coming into ARCore (it was during Huawei P30 presentation in Paris). The ToF sensor runs on the same framerate like RGB camera and it can detect objects in range 1cm-4m. I expect that it will be introduced on Google IO on the 7th May.
@Thaina even stereo cameras cannot detect white walls like ToF camera. Anyway I made wall detection which is working on all Android ARCore devices: https://www.linkedin.com/feed/update/urn:li:ugcPost:6512251490525020160 (You can download it from Unity Asset Store).
But what makes me more wrinkles is that Google is decreasing amount of support here (there are topics unanswered very long). This is the same behaviour they did shortly before ending with Google Tango (which was in summer 2017).
Tango was killed after Apple introduced ARkit. Will be ARCore killed because Microsoft introduced Hololens 2?
@lvonasek That is really good to know, thanks!
Also there are two software only based approaches by Facebook research and Google that look really promising and are supposed to work at camera framerates.
The Facebook approach is open source but only a slightly crippled Python reference version. The parts that make it real-time are left out of the Python code. By default it takes minutes per frame. I was able to optimize it and best i could achieve was 12 FPS at very low res with the Python code, but it was proof to me that it can work.
https://github.com/facebookresearch/AR-Depth
The Google approach is similar but no open-source code available. Would be great if they support tof devices and then use the software approach on other devices later.
I totally agree with @lvonasek , can you link some source for the Huawei statement?
For me and people i know ARCore is only something to play with. The capabilities are that limited it can be used for many products, even more real depth data via TOF was more meaningfull. As they announced ARCore i hope it will be the best out of all worlds, but as it seems we cannot expect big changes in the future to this topic.
@lvonasek It's not that I don't wish that all the phone would have ToF depth sensor. I just say that it was not practical to have it in most customer phone. Only some phone will add this feature as a bonus. It would less likely to have wide adoption
Surely it will eliminate almost all of AR problem but it don't have much demand unlike multi camera that used in normal photography. So I think we better not put our hope for ToF and should put more effort in Dual Camera unless there was a demand for ToF in other field (for what? I don't know, interior mapping maybe?)
@SimonScholl - https://youtu.be/2xMa3UZbRUU at 1:36.08
@Thaina - I do not believe in massive adoption of ToF sensor or stereo camera (of course there are more phones with stereo than ToF). Anyway I expect that AR on phones have no long future but that's just my opinion.