Out of Bounds issue when using own feature extractor

[half-potato]

I am trying to use hloc to test my own feature extractor but I'm running into a out of bounds issue in localize_sfm.main.

Here is the error. There are a couple print statements in there, so the lines are not correct.

query/day/nexus4/IMG_20140520_211432.jpg db/1420.jpg query-day-nexus4-IMG_20140520_211432.jpg_db-1420.jpg
[184] valid
[22] matches (0,)


---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-28-af749012d54f> in <module>
      6     outputs / f"{feature_conf['output']}_{matcher_conf['output']}_{loc_pairs.stem}.h5",
      7     results,
----> 8     covisibility_clustering=False)  # not required with SuperPoint+SuperGlue

/app/hloc/localize_sfm.py in main(reference_sfm, queries, retrieval, features, matches, results, ransac_thresh, covisibility_clustering)
    166             ret, mkpq, mp3d, mp3d_ids, num_matches, map_ = pose_from_cluster(
    167                 qname, qinfo, db_ids, db_images, points3D,
--> 168                 feature_file, match_file, thresh=ransac_thresh)
    169             # logging.info(f'# inliers: {ret["num_inliers"]}')
    170 

/app/hloc/localize_sfm.py in pose_from_cluster(qname, qinfo, db_ids, db_images, points3D, feature_file, match_file, thresh)
     63         print(valid, "valid")
     64         print(matches[valid], "matches", points3D_ids.shape)
---> 65         print(points3D_ids[matches[valid]], "points")
     66         valid = valid[points3D_ids[matches[valid]] != -1]
     67         num_matches += len(valid)

IndexError: index 22 is out of bounds for axis 0 with size 0

It may be possible that some of the files in the dataset are not the right ones, but I don't think that this is the case. A lot of the other matches have 0 valid, so it might be the case that my extractor is so bad that it creates an edge case.

[sarlinpe]

points3D_ids, defined as db_images[db_id].point3D_ids, is empty, so this database image has no detected keypoints in the SfM model. This is inconsistent with the match file, which expects some keypoints (at least 23 here). Both the match file and the SfM model are computed from the feature file.

1. Are you sure that you used the right SfM model and not an empty model?
2. Does the localization run smoothly for other query images or is it crashing on the first one?
3. Could you check the feature file for this database image?

features = h5py.open('path/to/your/feature/file.h5', 'r')
print(features['db/1420.jpg']['keypoints'].__array__().shape)

[half-potato]

sfm_superpoint+superglue:
total 67M
-rwxrwxrwx 1 root root 67M Sep 11 14:42 database.db
drwxrwxrwx 1 root root 368 Sep 11 14:42 model

sfm_superpoint+superglue/model:
total 83M
-rwxrwxrwx 1 root root 237K Sep 11 14:42 cameras.bin
-rwxrwxrwx 1 root root  82M Sep 11 14:42 images.bin
-rwxrwxrwx 1 root root 713K Sep 11 14:42 points3D.bin

[09/11/2020 21:41:49 INFO] Performing geometric verification of the matches...
[09/11/2020 21:42:02 INFO] Running the triangulation...
[09/11/2020 21:42:02 INFO] colmap point_triangulator --database_path outputs/aachen/sfm_superpoint+superglue/database.db --image_path datasets/aachen/images/images_upright --input_path outputs/aachen/sfm_empty --output_path outputs/aachen/sfm_superpoint+superglue/model --Mapper.ba_refine_focal_length 0 --Mapper.ba_refine_principal_point 0 --Mapper.ba_refine_extra_params 0
[09/11/2020 21:42:51 INFO] Statistics:
{'mean_reproj_error': 1.455615,
 'mean_track_length': 3.824187,
 'num_observations': 34215,
 'num_observations_per_image': 7.905499,
 'num_reg_images': 4328,
 'num_sparse_points': 8947}

I printed the reference_sfm and it was correct: reference_sfm: outputs/aachen/sfm_superpoint+superglue/model 2. Here I have included the full output of localize_sfm after adding print(valid, matches.shape, pair) to pose_from_cluster and print(qname, qinfo) to main.

reference_sfm: outputs/aachen/sfm_superpoint+superglue/model
datasets/aachen/queries
[09/11/2020 22:43:06 INFO] Importing 824 queries in day_time_queries_with_intrinsics.txt
[09/11/2020 22:43:06 INFO] Importing 98 queries in night_time_queries_with_intrinsics.txt
[09/11/2020 22:43:06 INFO] Reading 3D model...
[09/11/2020 22:43:09 INFO] Starting localization...

  0%|          | 1/922 [00:00<00:31, 29.64it/s]

query/day/nexus4/IMG_20140520_182846.jpg ('SIMPLE_RADIAL', 1600, 1200, array([ 1.46920e+03,  8.00000e+02,  6.00000e+02, -3.53019e-02]))
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-1410.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-45.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-452.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-591.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-1408.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-1412.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-53.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-312.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-451.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-448.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-447.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-42.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-1406.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-1848.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-48.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-1880.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-590.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-1847.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-1404.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-313.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-1436.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-592.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-315.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-316.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-1452.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-1300.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-1398.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-314.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-449.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-1298.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-317.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-1418.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-1889.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-1878.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-41.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-38.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-56.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-51.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-50.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-46.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-1396.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-446.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-54.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-1825.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-57.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-453.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-1826.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-1414.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-1874.jpg
[] (2599,) query-day-nexus4-IMG_20140520_182846.jpg_db-1881.jpg
query/day/nexus4/IMG_20140520_211432.jpg ('SIMPLE_RADIAL', 1600, 1200, array([ 1.46920e+03,  8.00000e+02,  6.00000e+02, -3.53019e-02]))
[] (1852,) query-day-nexus4-IMG_20140520_211432.jpg_db-52.jpg
[] (1852,) query-day-nexus4-IMG_20140520_211432.jpg_db-1520.jpg
[] (1852,) query-day-nexus4-IMG_20140520_211432.jpg_db-54.jpg
[] (1852,) query-day-nexus4-IMG_20140520_211432.jpg_db-50.jpg
[] (1852,) query-day-nexus4-IMG_20140520_211432.jpg_db-1872.jpg
[] (1852,) query-day-nexus4-IMG_20140520_211432.jpg_db-51.jpg
[] (1852,) query-day-nexus4-IMG_20140520_211432.jpg_db-1436.jpg
[] (1852,) query-day-nexus4-IMG_20140520_211432.jpg_db-1819.jpg
[] (1852,) query-day-nexus4-IMG_20140520_211432.jpg_db-1825.jpg
[] (1852,) query-day-nexus4-IMG_20140520_211432.jpg_db-1873.jpg
[184] (1852,) query-day-nexus4-IMG_20140520_211432.jpg_db-1420.jpg


---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-20-af749012d54f> in <module>
      6     outputs / f"{feature_conf['output']}_{matcher_conf['output']}_{loc_pairs.stem}.h5",
      7     results,
----> 8     covisibility_clustering=False)  # not required with SuperPoint+SuperGlue

/app/hloc/localize_sfm.py in main(reference_sfm, queries, retrieval, features, matches, results, ransac_thresh, covisibility_clustering)
    165             ret, mkpq, mp3d, mp3d_ids, num_matches, map_ = pose_from_cluster(
    166                 qname, qinfo, db_ids, db_images, points3D,
--> 167                 feature_file, match_file, thresh=ransac_thresh)
    168             # logging.info(f'# inliers: {ret["num_inliers"]}')
    169 

/app/hloc/localize_sfm.py in pose_from_cluster(qname, qinfo, db_ids, db_images, points3D, feature_file, match_file, thresh)
     61         valid = np.where(matches > -1)[0]
     62         print(valid, matches.shape, pair)
---> 63         valid = valid[points3D_ids[matches[valid]] != -1]
     64         num_matches += len(valid)
     65 

IndexError: index 22 is out of bounds for axis 0 with size 0

>>> features = h5py.File("feats-cpainted-n4096-r1024.h5", "r")
>>> print(features['db/1420.jpg']['keypoints'].__array__().shape)
(1515, 2)

Extra: Here is a visualization of the points. It appears as though my descriptors are buggy. I don't think they should be this bad.

[half-potato]

I just adapted the superpoint code to use my network and that fixed the issue. I don't want to close the issue because the bug isn't necessarily fixed, but you can if you want to. Edit: Ignore me it's still an issue. Just need to fix my model I guess.

[sarlinpe]

There is probably an issue with your descriptors since the SfM model you show stats for has very few points.

This should however not produce the crash mentioned above. I am sure that your model is still inconsistent with your feature file. Try out:

from hloc.utils.read_write_model import read_model
cameras, images, points3D = read_model('path/to/sfm/model', ext='.bin')
name2id = {images[i].name: i for i in images.keys()}
point3D_ids = images[name2id['db/1420.jpg']].point3D_ids
print(point3D_ids.shape)

If that is different than 1515 then the two files are inconsistent. The error message is indeed not very clear, I'll add some checks to ensure consistency. Thanks for bringing this up.

[half-potato]

from hloc.utils.read_write_model import read_model
cameras, images, points3D = read_model(reference_sfm / 'model', ext='.bin')
name2id = {images[i].name: i for i in images.keys()}
point3D_ids = images[name2id['db/1420.jpg']].point3D_ids
print(point3D_ids.shape)

(0,) They are indeed different. Will this be fixed when I fix the descriptors?

[sarlinpe]

No. As said above, your SfM model was built with a different feature file than the one that was used to generate the localization matches. Delete the SfM model and build it with the correct files.

[half-potato]

I have double checked file names and rerun all the computation, but I still get the same error. There isn't even another set of h5 files with which the SfM model could be built.

feats-cpainted-n4096-r1024.h5  
feats-cpainted-n4096-r1024_matches-superglue_pairs-db-covis20.h5  
feats-cpainted-n4096-r1024_matches-superglue_pairs-query-netvlad50.h5  
sfm_cpainted+superglue 
sfm_empty  
sfm_sift

What should I be looking at?

[sarlinpe]

Let's check all the images then:

features = h5py.File("feats-cpainted-n4096-r1024.h5", "r")
cameras, images, points3D = read_model('sfm_cpainted+superglue/model', ext='.bin')
for id_ in images:
    n_model = len(images[id_].point3D_ids)
    name = images[id_].name
    n_feats = features[name]['keypoints'].__array__().shape[0]
    print(name, id_, n_model, n_feats)

[half-potato]

So I'm not sure if the full output helps, but I captured it anyways. It appears that many of the images have the correct number of features, while some have none. https://pastebin.com/tnpH7smy

[sarlinpe]

Thanks.

1. It does not make any sense to match your new features with SuperGlue, unless you retrain it for these features. Please use the nearest neighbor matcher instead (with mutual check and potentially distance or ratio test).

2. It is likely that COLMAP does not bother adding the keypoints to the model if an image is not registered successfully. This is possibly due to weak descriptors or to 1. This edge case should still not lead to a crash - do you mind sending me your feature files + matches if they are not too large? My email is on my website.

[half-potato]

After switching to the NN matcher, the localization doesn't fail.

[half-potato]

Sorry, been struggling to get these feature files to you. The only thing I can think of is hosting it on AWS, which I'm still getting around to.