keras-team
Implement Depth Estimation Pipeline
Depth estimation is the process of determining the distance of objects from a viewpoint, which is crucial for various applications in computer vision, including autonomous driving, augmented reality, and 3D reconstruction..
Sample Input and Output:
Sample images (up) and depth annotation (down).
Applications of Depth Estimation:
- Estimation of Volumetric Information: Depth estimation models help study the volumetric formation of objects in images, crucial for computer graphics.
- 3D Representation: Depth estimation enables the development of 3D representations from 2D images.
- Augmented Reality: Depth estimation ensures accurate object placement and perspective calibration in AR applications.
- Robotics and Object Trajectory Estimation: Depth information helps estimate the motion trajectory of objects in 3D space.
- Haze and Fog Removal: Depth estimation aids in removing haze and fog by understanding how they affect distant objects.
- Portrait Mode: Depth-based blur in portrait mode enhances the focus on subjects and creates appealing background effects.
Depth Estimation Subtasks:
There are two depth estimation subtasks.
Absolute depth estimation: Absolute (or metric) depth estimation aims to provide exact depth measurements from the camera. Absolute depth estimation models output depth maps with real-world distances in meter or feet.
Relative depth estimation: Relative depth estimation aims to predict the depth order of objects or points in a scene without providing the precise measurements.
Depth Estimation Models
-
Depth Anything Downloads last month on huggingface: 592,379 Backbone: ResNet and Convolution based Cited by: 744 Published on: April 7, 2024 Paper: https://arxiv.org/pdf/2401.10891 Model card: https://huggingface.co/depth-anything/Depth-Anything-V2-Small-hf
-
DPT by Intel Labs research Downloads last month on huggingface: 162,385 Backbone: Convolution and VIT based Cited By: 2151 Released on: Mar 24, 2021 Paper: https://arxiv.org/pdf/2103.13413 Model card: https://huggingface.co/Intel/dpt-large
-
Depth Crafter by Tencent AI Lab Downloads last month on huggingface: 362,047 Backbone: UNet, CLIP and VAE based Cited By: 41 Released on: Nov 27, 2024 Paper: https://arxiv.org/pdf/2409.02095 Model card: https://huggingface.co/tencent/DepthCrafter
-
Depth Pro by Apple Downloads last month on huggingface: 51,653 Backbone: VIT Based Cited by: 54 Released on: Oct 2, 2024. Paper: https://arxiv.org/pdf/2410.02073 Model card: https://huggingface.co/apple/DepthPro-hf
Why will it be a good addition?
The backbone of above models already exist in keras-hub and above models will also be compatible with keras-hub structure and API, so it will be easy to add them. The pipeline will also follow the same workflow as other pipelines of image classifcation, object detection, etc. Also, above applications are quite crucial in daily life and this pipeline will help users perform above tasks through keras-hub with greater ease.
I'd love to hear comments from the community. FYI @divyashreepathihalli @mattdangerw @sineeli
depthanything is great! would you like to contribute this to KerasHub?
depthanything is great! would you like to contribute this to KerasHub?
Yes, I'd be happy to take that, starting from the depth estimation pipeline. Do we plan to add other famous depth estimation models(as listed above) in the future as well?
Any progress on this? Am hoping to incorporate the depth anything model into a keras training pipeline.
Any progress on this? Am hoping to incorporate the depth anything model into a keras training pipeline.
I'm currently working on another PR. Then, I've planned to implement depth estimation pipeline with depthanything model. I'd really appreciate that you contribute a model with me.
@openSourcerer9000 you can work on this one as I am currently not pursuing this .