Problem when training xfeat+lightglue

[limymy]

Thanks for open-sourcing this awesome work! I've successfully trained xfeat using your code. I'm running into some issues trying to train lightglue with xfeat as an extractor, following your instructions(https://github.com/verlab/accelerated_features/issues/67#issuecomment-2440035284). The results are pretty bad when I evaluate with gluefactory. After pre-training on the homography dataset:

best threshold = 2.5
{'H_error_dlt@1px': 0.0052,
 'H_error_dlt@3px': 0.0284,
 'H_error_dlt@5px': 0.0529,
 'H_error_ransac@1px': 0.265,
 'H_error_ransac@3px': 0.546,
 'H_error_ransac@5px': 0.6666,
 'H_error_ransac_mAA': 0.4925333333333333,
 'mH_error_dlt': 61.393,
 'mH_error_ransac': 0.994,
 'mnum_keypoints': 512.0,
 'mnum_matches': 244.0,
 'mprec@1px': 0.205,
 'mprec@3px': 0.505,
 'mransac_inl': 118.0,
 'mransac_inl%': 0.498}

Here is my conf file:

data:
    name: homographies
    data_dir: revisitop1m
    train_size: 150000
    val_size: 2000
    batch_size: 128
    num_workers: 16
    homography:
        difficulty: 0.5
        max_angle: 45
    photometric:
        name: lg
model:
    name: two_view_pipeline
    extractor:
        name: extractors.xfeat
        max_num_keypoints: 512
        detection_threshold: 0.05
        trainable: False
    ground_truth:
        name: matchers.homography_matcher
        th_positive: 3
        th_negative: 3
    matcher:
        name: matchers.lightglue
        filter_threshold: 0.1
        flash: true
        checkpointed: true
        input_dim: 64
        descriptor_dim: 96
        n_layers: 6
        num_heads: 1
train:
    seed: 0
    epochs: 40
    log_every_iter: 100
    eval_every_iter: 500
    lr: 1e-4
    lr_schedule:
        start: 20
        type: exp
        on_epoch: true
        exp_div_10: 10
    plot: [5, 'gluefactory.visualization.visualize_batch.make_match_figures']
benchmarks:
    hpatches:
      eval:
        estimator: poselib
        ransac_th: [0.5, 1.0, 1.5, 2.0, 2.5, 3.0]

And I utilized the DetectAndCompute pipeline from the provided source code as the _forward function within my extractor/xfeat implementation.

Could you share your modified conf file and extractor/xfeat.py code file? Seeing how you've set up the xfeat extractor would be really useful.

[ZhouShunlong]

Mind sharing what GPUs you're using?

[1303463822]

Hello, are you still training xfeat lightglue? I’ve also been working on it recently. Could you share some information about the content in 'extractor/xfeat.py'?

[1303463822]

When training the LightGlue model based on Xfeat, I encounter the 'Detected NAN, skipping iteration 1249' issue.