Orientation issue with equirectangular images

[joeboers]

I'm trying to run the reconstruction on a set of sequential equirectangular images recorded with a GoPro MAX on top of a car. No matter what I do, halfway the segment the orientation seems to go upside down during a turn. The images have been pre-filtered so that each is about 1 meter apart.

I thought it was just the frontal direction of the camera being changed as is apparent in this plot:

Note: red dots are the camera origins, green lines/dots the bearing for the middle of the image

Where the orientation of the camera relative to the driving suddenly swaps to the other side, even though it is fixed in reality.

I've also plotted the Euler angles calculated from the rotation matrix and it seems the camera orientation gradually flips upside down during the turn:

This is the original gps trace:

I've tried setting the orientation prior to horizontal, no_roll or naive, but same issue. With or without GPS data, 4k or 16k features, increasing feature image size, increased matches, local_bundle_radius,... none of it seems to have any effect.

Any idea to why this is happening? or where I can look to debug the issue?

This is the full config that I use:

{
    'processes': 32, 
    'matching_time_neighbors': 2,
    'matching_gps_neighbors': 0,
    'matching_gps_distance': 0,
    
    'local_bundle_radius': 1,
    'use_altitude_tag': 'no',
    'align_orientation_prior': 'horizontal',
}

In the exif_overrides I've atm only configured the capture_time and the orientation (equal to 1).

[joeboers]

I just noticed there is an elevated speedbump in the turn, which might complicate things

[joeboers]

It seems the viewing direction of the camera is pointed downwards somehow even though the altitude differences and locations are quite correct (I added the altitude gps exif tag again here)

[joeboers]

Weird, the camera directions in the viewer look correct.

Can someone verify wether I'm calculating the direction correctly?

data = dataset.DataSet(dataset_path)

reconstruction = data.load_reconstruction(filename='reconstruction.json')[0]

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

for shot_id, shot in reconstruction.shots.items():    
    origin = shot.pose.get_origin()
    ax.scatter(origin[0], origin[1], origin[2], c='green', linewidth=0, marker='o')
    
    # Calculate direction line for 10 meters forward
    direction_end = shot.pose.transform_inverse([0,0,10])
    line = np.vstack([origin.reshape((1, -1)), direction_end.reshape((1, -1))])
    
    ax.plot(line[:, 0], line[:, 1], line[:, 2], c='red', linewidth=0.5)

[athamour1]

i have a similar project how did you reconstract equirectangular images with opensfm, i am new to this i dont know much.

[EliDavis3D]

For your video, do you have the world lock turned on from the videoplayer export? Are you using masks?

[chris-cooper]

In case it helps anyone else who ends up here, make sure you also set align_method:

align_method: orientation_prior     # Method to use for aligning images. Can be: [orientation_prior, auto, naive]
align_orientation_prior: horizontal # Orientation prior to use for alignment. Can be: [horizontal, vertical, auto]

• https://github.com/mapillary/OpenSfM/blob/8887d336cdc305427d59d02a96ef9396aa197ad3/opensfm/align.py#L238-L251
• https://github.com/mapillary/OpenSfM/blob/8887d336cdc305427d59d02a96ef9396aa197ad3/opensfm/align.py#L82-L88