AliceVision icon indicating copy to clipboard operation
AliceVision copied to clipboard

Add software to filter SfM data and to create landmarks-based masking for depth map computation

Open almarouk opened this issue 2 years ago • 0 comments

Linked to https://github.com/alicevision/Meshroom/pull/2211

Description

This PR introduces the following two main features:

  • Depth maps computation can be masked using landmarks from SfM data.
  • SfM data can be filtered to efficiently be used for depth maps masking.

The SfM data filtering supports the following features (the parameters are independent):

  • Filtering landmarks over 3 steps:
    • Step 1: Remove landmarks that have an insufficient number of landmarks. Parameters: minNbObservationsPerLandmark
    • Step 2: Remove landmarks with dissimilar observations compared to their neighbors in 3D. Parameters: nbNeighbors3D, minNbObservationsPerLandmark
    • Step 3: Remove landmarks with worse resolution than their neighbors in 3D. The resolution is based on pixel size. Parameters: radiusScale, useFeatureScale
  • Filtering observations to select best observations per landmark and to ensure consistency with neighbors in 3D. The observations are scored in function of distance between corresponding camera centers and landmarks positions. The scores are iteratively propagated between neighboring landmarks in 3D. Parameters: maxNbObservationsPerLandmark, propagationEnabled, nbNeighbors3D, nbIterations, dampingEnabled, dampingFactor
  • Filtering observations (and eventually landmarks) to ensure non-sparsity in 2D projection of landmarks separately in each view. Mask radius for projected landmarks per view is estimated and reported in an output text file (outputRadiiFile). Parameters: nbNeighbors2D, percentile, maskRadiusThreshold

Future Work

  • [ ] fix duplicate landmarks' biased influence.
  • [ ] take into consideration the angle of observation when scoring the observations in addition to the distance to camera.
  • [ ] include blur estimation.
  • [ ] have a more complex relative estimation for neighborhoods (e.g. elliptic neighborhood with orientation, based on landmarks segmentation/classification for surface information...).
  • [ ] verify if modifications are required in case of multiple capturing devices.

almarouk avatar Sep 06 '23 14:09 almarouk