Some doubts regarding view projection and field of view.

[coreqode]

Hello. Thanks for an amazing work. I was trying this method on custom data, however there is a lot of noise in the geometry (even if I initialise point cloud using a mesh). However in case of synthetic data there is no noise in the optimised point clouds.

On looking the code carefully, I realized that getProjectionMatrix and focal2fov are only valid when principal point cx and cy are center of the image pixels. If we get unbalanced (different from mid point) principal points from the calibration, we might get different fovs and view transform matrix?

[qhdqhd]

same question

[deepshwang]

I'm trying to figure this out from this link

https://strawlab.org/2011/11/05/augmented-reality-with-OpenGL/

but there's no explanation on its derivation.

Any ideas on making an actual implementation considering the z_sign? (which I failed to understand why it's only multiplied on P[3, 2] and P[2, 2] only)

[f-dy]

+1: principal point (PP) is ignored, leading to wrong geometry.

One must either fix the PP at SfM time (default for COLMAP, not the default for other SfM software) or translate images during undistortion so that the PP is at the center (not possible with COLMAP afaik, but rather simple to do with OpenCV).

The real code fix would be to handle the PP in the camera model, but it involves more changes.

In the mean time, the code should check here that the PP is at the center, else crash with an error.

[Ryoto-Kato]

I had also encountered this issue due to (cx, cy) != (width/2, height/2) because getProjectionMatrix() assumes (cx, cy) == (width/2, height/2)

In order to adapt to the non-zero principle point offset, you need to shift the frame window depending on how much the principle point deviates from the center of the near plane (the offset at the near plane).

How about using following getProjectionMatrixShift() instead of getProjectionMatrix()??

def getProjectionMatrixShift(znear, zfar, focal_x, focal_y, cx, cy, width, height, fovX, fovY):
    tanHalfFovY = math.tan((fovY / 2))
    tanHalfFovX = math.tan((fovX / 2))

    # the origin at center of image plane
    top = tanHalfFovY * znear
    bottom = -top
    right = tanHalfFovX * znear
    left = -right

    # shift the frame window due to the non-zero principle point offsets
    offset_x = cx - (width/2)
    offset_x = (offset_x/focal_x)*znear
    offset_y = cy - (height/2)
    offset_y = (offset_y/focal_y)*znear

    top = top + offset_y
    left = left + offset_x
    right = right + offset_x
    bottom = bottom + offset_y

    P = torch.zeros(4, 4)

    z_sign = 1.0

    P[0, 0] = 2.0 * znear / (right - left)
    P[1, 1] = 2.0 * znear / (top - bottom)
    P[0, 2] = (right + left) / (right - left)
    P[1, 2] = (top + bottom) / (top - bottom)
    P[3, 2] = z_sign
    P[2, 2] = z_sign * zfar / (zfar - znear)
    P[2, 3] = -(zfar * znear) / (zfar - znear)
    return P

[hariharan1412]

could you please tell me how to get 'focal_x' and 'focal_y' ?

[Dargonxzy]

could you please tell me how to get 'focal_x' and 'focal_y' ?

intrinsics

[Dargonxzy]

I had also encountered this issue due to (cx, cy) != (width/2, height/2) because getProjectionMatrix() assumes (cx, cy) == (width/2, height/2)

In order to adapt to the non-zero principle point offset, you need to shift the frame window depending on how much the principle point deviates from the center of the near plane (the offset at the near plane).

How about using following getProjectionMatrixShift() instead of getProjectionMatrix()??

def getProjectionMatrixShift(znear, zfar, focal_x, focal_y, cx, cy, width, height, fovX, fovY):
    tanHalfFovY = math.tan((fovY / 2))
    tanHalfFovX = math.tan((fovX / 2))

    # the origin at center of image plane
    top = tanHalfFovY * znear
    bottom = -top
    right = tanHalfFovX * znear
    left = -right

    # shift the frame window due to the non-zero principle point offsets
    offset_x = cx - (width/2)
    offset_x = (offset_x/focal_x)*znear
    offset_y = cy - (height/2)
    offset_y = (offset_y/focal_y)*znear

    top = top + offset_y
    left = left + offset_x
    right = right + offset_x
    bottom = bottom + offset_y

    P = torch.zeros(4, 4)

    z_sign = 1.0

    P[0, 0] = 2.0 * znear / (right - left)
    P[1, 1] = 2.0 * znear / (top - bottom)
    P[0, 2] = (right + left) / (right - left)
    P[1, 2] = (top + bottom) / (top - bottom)
    P[3, 2] = z_sign
    P[2, 2] = z_sign * zfar / (zfar - znear)
    P[2, 3] = -(zfar * znear) / (zfar - znear)
    return P

I would like to ask in this case have you made any changes to the comoputeConv2D?

[whu-lyh]

Should the $J$ matrix at computeCov2D also be adjusted?

// Forward version of 2D covariance matrix computation
__device__ float3 computeCov2D(const float3& mean, float focal_x, float focal_y, float tan_fovx, float tan_fovy, const float* cov3D, const float* viewmatrix)
{
...
	glm::mat3 J = glm::mat3(
		focal_x / t.z, 0.0f, -(focal_x * t.x) / (t.z * t.z),
		0.0f, focal_y / t.z, -(focal_y * t.y) / (t.z * t.z),
		0, 0, 0);
...
}