SFM Stage Error: cudaMemcpyToSymbol failed

[cturner8725]

Hello All,

I am running into an error with gpu proccessing during the SFM stage with gives me the following message

“cudaMemcpyToSymbol failed for Gauss kernel initialization: unrecognized error code”

I am using the Windows Native WebODM version 2.7.0 on Windows 11 24H2 with all the latest drivers. GPU is a RTX 5080. When I restart the task, the SFM stage will proceed using my CPU. When the MVS stage begins, it will process on my GPU without error. This has been replicated multiple times with different data sets.

Can someone help me find a solution to this?

Photos and Tasks Output Exports: https://drive.google.com/drive/folders/1TgJm_hz7dHh6OIXDHHVVz6A11FnGtoQc?usp=drive_link

Parameters: Default; Resize Images 2048

[INFO] Initializing ODM 3.5.4 - Fri Feb 21 19:03:40 2025 [INFO] ============== [INFO] 3d_tiles: False [INFO] align: None [INFO] auto_boundary: True [INFO] auto_boundary_distance: 0 [INFO] bg_removal: False [INFO] boundary: {} [INFO] build_overviews: False [INFO] camera_lens: auto [INFO] cameras: {} [INFO] cog: True [INFO] copy_to: None [INFO] crop: 3 [INFO] dem_decimation: 1 [INFO] dem_euclidean_map: False [INFO] dem_gapfill_steps: 3 [INFO] dem_resolution: 5 [INFO] dsm: True [INFO] dtm: False [INFO] end_with: odm_postprocess [INFO] fast_orthophoto: False [INFO] feature_quality: high [INFO] feature_type: dspsift [INFO] force_gps: False [INFO] gcp: None [INFO] geo: None [INFO] gltf: True [INFO] gps_accuracy: 3 [INFO] ignore_gsd: False [INFO] matcher_neighbors: 0 [INFO] matcher_order: 0 [INFO] matcher_type: flann [INFO] max_concurrency: 16 [INFO] merge: all [INFO] mesh_octree_depth: 11 [INFO] mesh_size: 200000 [INFO] min_num_features: 10000 [INFO] name: 66102c90-90ea-458a-ac16-388a2606ae2a [INFO] no_gpu: False [INFO] optimize_disk_space: False [INFO] orthophoto_compression: DEFLATE [INFO] orthophoto_cutline: False [INFO] orthophoto_kmz: False [INFO] orthophoto_no_tiled: False [INFO] orthophoto_png: False [INFO] orthophoto_resolution: 5 [INFO] pc_classify: False [INFO] pc_copc: False [INFO] pc_csv: False [INFO] pc_ept: True [INFO] pc_filter: 5 [INFO] pc_las: False [INFO] pc_quality: medium [INFO] pc_rectify: False [INFO] pc_sample: 0 [INFO] pc_skip_geometric: False [INFO] primary_band: auto [INFO] project_path: C:\WebODM\resources\app\apps\NodeODM\data\66102c90-90ea-458a-ac16-388a2606ae2a [INFO] radiometric_calibration: none [INFO] rerun: None [INFO] rerun_all: False [INFO] rerun_from: None [INFO] rolling_shutter: False [INFO] rolling_shutter_readout: 0 [INFO] sfm_algorithm: incremental [INFO] sfm_no_partial: False [INFO] skip_3dmodel: False [INFO] skip_band_alignment: False [INFO] skip_orthophoto: False [INFO] skip_report: False [INFO] sky_removal: False [INFO] sm_cluster: None [INFO] sm_no_align: False [INFO] smrf_scalar: 1.25 [INFO] smrf_slope: 0.15 [INFO] smrf_threshold: 0.5 [INFO] smrf_window: 18.0 [INFO] split: 999999 [INFO] split_image_groups: None [INFO] split_overlap: 150 [INFO] texturing_keep_unseen_faces: False [INFO] texturing_single_material: False [INFO] texturing_skip_global_seam_leveling: False [INFO] tiles: False [INFO] use_3dmesh: False [INFO] use_exif: False [INFO] use_fixed_camera_params: False [INFO] use_hybrid_bundle_adjustment: False [INFO] video_limit: 500 [INFO] video_resolution: 4000 [INFO] ============== [INFO] Running dataset stage [INFO] Loading dataset from: C:\WebODM\resources\app\apps\NodeODM\data\66102c90-90ea-458a-ac16-388a2606ae2a\images [INFO] Loading 130 images [INFO] Wrote images database: C:\WebODM\resources\app\apps\NodeODM\data\66102c90-90ea-458a-ac16-388a2606ae2a\images.json [INFO] Found 130 usable images [INFO] Finished dataset stage [INFO] Running split stage [INFO] Normal dataset, will process all at once. [INFO] Finished split stage [INFO] Running merge stage [INFO] Normal dataset, nothing to merge. [INFO] Finished merge stage [INFO] Running opensfm stage [INFO] Maximum photo dimensions: 1280px [INFO] Photo dimensions for feature extraction: 1280px [INFO] CUDA drivers detected [INFO] Altitude data detected, enabling it for GPS alignment [INFO] [‘use_exif_size: no’, ‘flann_algorithm: KDTREE’, ‘feature_process_size: 1280’, ‘feature_min_frames: 10000’, ‘processes: 16’, ‘matching_gps_neighbors: 0’, ‘matching_gps_distance: 0’, ‘matching_graph_rounds: 50’, ‘optimize_camera_parameters: yes’, ‘reconstruction_algorithm: incremental’, ‘undistorted_image_format: tif’, ‘bundle_outlier_filtering_type: AUTO’, ‘sift_peak_threshold: 0.066’, ‘align_orientation_prior: vertical’, ‘triangulation_type: ROBUST’, ‘retriangulation_ratio: 2’, ‘matcher_type: FLANN’, ‘feature_type: DSPSIFT’, ‘use_altitude_tag: yes’, ‘align_method: auto’, ‘local_bundle_radius: 0’] [INFO] Wrote reference_lla.json [INFO] running “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm” detect_features “C:\WebODM\resources\app\apps\NodeODM\data\66102c90-90ea-458a-ac16-388a2606ae2a\opensfm” d:\a\odm\odm\superbuild\build\pypopsift_deps\popsift-src\src\popsift\gauss_filter.cu:245 cudaMemcpyToSymbol failed for Gauss kernel initialization: unrecognized error code

[ghost]

I'm also getting this same issue with ODM (non-web version). Looks very related to this: https://github.com/alicevision/popsift/issues/160, and might imply that either the popsift dependency is out of date, or else not compiled for your graphics card. Do you know the CC code for your card? You can find it here: https://developer.nvidia.com/cuda-gpus

[cptcitrus]

I've just had this error on a new machine with a RTX 5080, CC 7.5. Error message attached. Desktop version of ODM.

Edit: I've now replicated it across three different imagery collections from two different UAVs. Will test on another machine next.

message.txt

[cturner8725]

I'm also getting this same issue with ODM (non-web version). Looks very related to this: alicevision/popsift#160, and might imply that either the popsift dependency is out of date, or else not compiled for your graphics card. Do you know the CC code for your card? You can find it here: https://developer.nvidia.com/cuda-gpus

I have a RTX 5080 like cptcitrus, so CC 7.5 as well. Looks like that code is also used for RTX 4000 and 3000 series as well. So, I would think this issue would be more prevalent if the CC code was the issue, right?

Edit: I've now replicated it across three different imagery collections from two different UAVs. Will test on another machine next.

I tested this across multiple data sets as well with different image counts and varied reduced image sizes. Same error. Have you tried restarting the task to see if the MVS stage uses your GPU?

[RomanIvn]

Same with WebODM web using DigitalOcean GPU droplet & 1XH100 running NodeODM in docker.

[INFO] DTM is turned on, automatically turning on point cloud classification [INFO] Initializing ODM 3.5.5 - Sat Mar 15 22:03:28 2025 [INFO] ============== [INFO] 3d_tiles: True [INFO] align: None [INFO] auto_boundary: False [INFO] auto_boundary_distance: 0 [INFO] bg_removal: False [INFO] boundary: {} [INFO] build_overviews: False [INFO] camera_lens: auto [INFO] cameras: {} [INFO] cog: True [INFO] copy_to: None [INFO] crop: 3 [INFO] dem_decimation: 1 [INFO] dem_euclidean_map: False [INFO] dem_gapfill_steps: 3 [INFO] dem_resolution: 2.0 [INFO] dsm: True [INFO] dtm: True [INFO] end_with: odm_postprocess [INFO] fast_orthophoto: False [INFO] feature_quality: ultra [INFO] feature_type: dspsift [INFO] force_gps: False [INFO] gcp: None [INFO] geo: None [INFO] gltf: True [INFO] gps_accuracy: 3 [INFO] ignore_gsd: False [INFO] matcher_neighbors: 0 [INFO] matcher_order: 0 [INFO] matcher_type: flann [INFO] max_concurrency: 60 [INFO] merge: all [INFO] mesh_octree_depth: 11 [INFO] mesh_size: 200000 [INFO] min_num_features: 10000 [INFO] name: 3dd1d6b7-3ce5-41f3-ac0f-a5ca4b54f643 [INFO] no_gpu: False [INFO] optimize_disk_space: False [INFO] orthophoto_compression: DEFLATE [INFO] orthophoto_cutline: False [INFO] orthophoto_kmz: False [INFO] orthophoto_no_tiled: False [INFO] orthophoto_png: False [INFO] orthophoto_resolution: 2.0 [INFO] pc_classify: True [INFO] pc_copc: False [INFO] pc_csv: False [INFO] pc_ept: True [INFO] pc_filter: 5 [INFO] pc_las: False [INFO] pc_quality: ultra [INFO] pc_rectify: False [INFO] pc_sample: 0 [INFO] pc_skip_geometric: False [INFO] primary_band: auto [INFO] project_path: /var/www/data/3dd1d6b7-3ce5-41f3-ac0f-a5ca4b54f643 [INFO] radiometric_calibration: none [INFO] rerun: None [INFO] rerun_all: False [INFO] rerun_from: ['opensfm', 'openmvs', 'odm_filterpoints', 'odm_meshing', 'mvs_texturing', 'odm_georeferencing', 'odm_dem', 'odm_orthophoto', 'odm_report', 'odm_postprocess'] [INFO] rolling_shutter: False [INFO] rolling_shutter_readout: 0 [INFO] sfm_algorithm: incremental [INFO] sfm_no_partial: False [INFO] skip_3dmodel: False [INFO] skip_band_alignment: False [INFO] skip_orthophoto: False [INFO] skip_report: False [INFO] sky_removal: False [INFO] sm_cluster: None [INFO] sm_no_align: False [INFO] smrf_scalar: 1.25 [INFO] smrf_slope: 0.15 [INFO] smrf_threshold: 0.5 [INFO] smrf_window: 18.0 [INFO] split: 999999 [INFO] split_image_groups: None [INFO] split_overlap: 150 [INFO] texturing_keep_unseen_faces: False [INFO] texturing_single_material: False [INFO] texturing_skip_global_seam_leveling: False [INFO] tiles: False [INFO] use_3dmesh: False [INFO] use_exif: False [INFO] use_fixed_camera_params: False [INFO] use_hybrid_bundle_adjustment: False [INFO] video_limit: 500 [INFO] video_resolution: 4000 [INFO] ============== [INFO] Running dataset stage [INFO] Loading dataset from: /var/www/data/3dd1d6b7-3ce5-41f3-ac0f-a5ca4b54f643/images [INFO] Loading images database: /var/www/data/3dd1d6b7-3ce5-41f3-ac0f-a5ca4b54f643/images.json [INFO] Found 45 usable images [INFO] Coordinates file already exist: /var/www/data/3dd1d6b7-3ce5-41f3-ac0f-a5ca4b54f643/odm_georeferencing/coords.txt [INFO] Model geo file already exist: /var/www/data/3dd1d6b7-3ce5-41f3-ac0f-a5ca4b54f643/odm_georeferencing/odm_georeferencing_model_geo.txt [INFO] Finished dataset stage [INFO] Running split stage [INFO] Normal dataset, will process all at once. [INFO] Finished split stage [INFO] Running merge stage [INFO] Normal dataset, nothing to merge. [INFO] Finished merge stage [INFO] Running opensfm stage [INFO] Maximum photo dimensions: 2048px [INFO] Photo dimensions for feature extraction: 2048px [INFO] nvidia-smi detected [INFO] Altitude data detected, enabling it for GPS alignment [INFO] ['use_exif_size: no', 'flann_algorithm: KDTREE', 'feature_process_size: 2048', 'feature_min_frames: 10000', 'processes: 60', 'matching_gps_neighbors: 0', 'matching_gps_distance: 0', 'matching_graph_rounds: 50', 'optimize_camera_parameters: yes', 'reconstruction_algorithm: incremental', 'undistorted_image_format: tif', 'bundle_outlier_filtering_type: AUTO', 'sift_peak_threshold: 0.066', 'align_orientation_prior: vertical', 'triangulation_type: ROBUST', 'retriangulation_ratio: 2', 'matcher_type: FLANN', 'feature_type: DSPSIFT', 'use_altitude_tag: yes', 'align_method: auto', 'local_bundle_radius: 0'] [INFO] Wrote reference_lla.json [INFO] running "/code/SuperBuild/install/bin/opensfm/bin/opensfm" detect_features "/var/www/data/3dd1d6b7-3ce5-41f3-ac0f-a5ca4b54f643/opensfm" /code/SuperBuild/build/pypopsift/_deps/popsift-src/src/popsift/gauss_filter.cu:245 cudaMemcpyToSymbol failed for Gauss kernel initialization: unrecognized error code

[cptcitrus]

The WebODM processing completed successfully when no-gpu flag was ticked. This is definitely a GPU issue. I can post the log if helpful but not today.

[ghost]

It looks like there's maybe two different issues going on here.

ODM uses this version of PopSift, which hasnt been updated in a hot minute, and is hard-coded to only compile for some specific architectures here:

https://github.com/uav4geo/popsift/blob/c9719924d6c6926ca9e443b2766c07c4f99e8177/cmake/ChooseCudaCC.cmake#L68

So when RomanIvn and I try to run this on Digital Ocean's GPU stuff, the application just doesnt handle the newer hardware. Adding 90, to that line I linked should hypothetically fix it for us two, though I've not gotten that to work just yet.

But yeah for cpcitrus and cturner8725, idk what's going on, popsift is very explicitly built for your cards. Do you guys get anything like a build log? There should be a line in there that says Compiling for CUDA CCs: 35;50;52;60;61;70;75;80;86

[cptcitrus]

Okay, I've identified the issue. I was looking at the wrong CC on NVIDIA's website (RTX 5000 vs Geforce RTX 5080). My compute score is 10.0, which is above the level popsift can support. Can you confirm this theory?

From popsift code:

  # Create a list of possible CCs for each host processor.
  # This may require tuning: CUDA cards exist in AIX machines with POWER CPUs,
  # it is possible that non-Tegra ARM systems exist as well.
  # For now, this is my best guess.
  #
  set(TEGRA_SUPPORTED_PROCESSORS "armv71;arm;aarch64")
  set(OTHER_SUPPORTED_PROCESSORS "i686;x86_64;AMD64")

  set(CC_LIST_BY_SYSTEM_PROCESSOR "")
  if(CMAKE_SYSTEM_PROCESSOR IN_LIST OTHER_SUPPORTED_PROCESSORS)
    list(APPEND CC_LIST_BY_SYSTEM_PROCESSOR "20;21;30;35;50;52;60;61;70;75;80;86")
  endif()
  if(CMAKE_SYSTEM_PROCESSOR IN_LIST TEGRA_SUPPORTED_PROCESSORS)
    list(APPEND CC_LIST_BY_SYSTEM_PROCESSOR "32;53;62;72")
  endif()
  if(NOT CC_LIST_BY_SYSTEM_PROCESSOR)
    message(FATAL_ERROR "Unknown how to build for ${CMAKE_SYSTEM_PROCESSOR}")
  endif()

[ghost]

yeah that looks right to me

Fix-wise: do you need need to use GPU acceleration? I might just skip it, at least for this specific stage. I don't think running SIFT on CPU is too expensive...

Also hey another Quinn in the wild!

[cptcitrus]

Another Quinn working in lidR even. We could start a club.

I think I will skip the GPU acceleration for now, which is too bad but not a big deal. Maybe popsift will be updated eventually.

[LightningStorm0]

Is there a simple way to skip GPU acceleration just for the SFM step to avoid this error?

[protoblade]

so longstory short - some modificatios on odm are required to run - in my case its successful by using a code that is not yet merged into the official popsift repo link and having cmake 3.24 (again from the same PR), running on the latest nvidia/cuda:12.9.1. Additionally the original repo has removed the hardcoded versions of the supported nvidia cc so this solves additional problems caused by the library.

PS. for anyone that cannot wait for the chain of events to happen this image was build with latest cuda libs and relatively new (1 year old) popsift and works on (at least) 5070TI

[Detritalgeo]

Yes, just happened with my Lenovo Legion Pro 7i with RTX5090. Failed with same error and restarted on CPU only. CPU usage is ~11%, it is an Intel Core Ultra 275HX with 24C

[MrMPod]

I have that issue too - GPU issue for RTX5xxx - Processing failed (4294967051)

After starting the processing, an error message “Processing failed (4294967051)” pops up, and the processing stops. Tested with default settings and high resolution. I know there is a way with use --no-gpu BUT it is soooooo loooong. With GPU and 591 photos it will take about 3 hours but with --no-gpu it is over 24 hours.

WebODM installation was done via a purchased installer.

[dburk1013]

I've been working with desktop WebODM (with the purchased installer) for about 2 hours now. So I'm really new. Forgive my asking this, but is the bottom line:

1. If I have an NVIDEA GeForce RTX 5060 video card, there is nothing I can do to use the gpu for processing?
2. If that's the case, is there anyone who is working on the fix for this? Can there ultimately be a fix for this?
3. Would an older, slower video card work?
4. I actually have an older PC with an older video card that I could dedicate to WebODM. Are there instructions for setting up one computer on my LAN to become a Processing Node for the WebODM running on my desktop? Or did I just spill the beans on how I really don't understand what's going on? (Be nice!)

[smathermather]

Hi @dburk1013 -- unfortunately, we don't have anything in the works yet to fix this issue. Older slower cards do work. Join us at community.opendronemap.org and we can help you sort through setting up a separate processing node.

[protoblade]

I've been working with desktop WebODM (with the purchased installer) for about 2 hours now. So I'm really new. Forgive my asking this, but is the bottom line:

1. If I have an NVIDEA GeForce RTX 5060 video card, there is nothing I can do to use the gpu for processing?
2. If that's the case, is there anyone who is working on the fix for this? Can there ultimately be a fix for this?
3. Would an older, slower video card work?

1. There is. kind of hacky but it works. take a look at my previous post. essentially what you need to do is replace the original image definition with this one and it works.
2. i hope there is another one. but i can share my changes if needed
3. anything older than 4xxx should work out of the box. if i remember correctly the cuda version was old enough to not support 4xxx as well....

[protoblade]

Yes, just happened with my Lenovo Legion Pro 7i with RTX5090. Failed with same error and restarted on CPU only. CPU usage is ~11%, it is an Intel Core Ultra 275HX with 24C

was this using the original image or my modified one ?

[MrMPod]

This has been going on for a while and there are new reports; it concerns the latest cards, so it might be worth giving this the right priority, because the software is excellent and it would be a shame to discourage new people who are already coming with 50xx cards