[Question about ViewMatrix Solving process]

[tomguluson92]

Hi, Thank you so much for your work in coming up with NeRF! I found a little confused when meeting with the process in solving the look-at view matrix. I use the LLFF flowers dataset in my test. The questions are listed in the code below, I sincerely hope you answer, thank you very much!

The puzzled point occurs due to the inconformity between code and the definition in LLFF. So I want some help here, thank you!

def viewmatrix(z, up, center):
    """
        right-handed coordinate system.
    """
    vec2 = normalize(z)
    vec1_avg = up
    # **fixme**
    # do I need to normalize the vec1_avg = normalize(up) 
    # as well as reform the 
    # vec2 as normalize(normalize(poses[:, :3, 2].sum(0)) - center)?
    vec0 = normalize(np.cross(vec1_avg, vec2))
    vec1 = normalize(np.cross(vec2, vec0))
    m = np.stack([vec0, vec1, vec2, center], 1)
    # m (3, 4)
    return m

def poses_avg(poses):
    """
        input:  poses(N, 3, 5)
        output: c2w  (N, 3, 5)

        I assume:
        Last dim(4) is height, width, and focal length.
        Dim (3) is the center (Could I regard it as the eye position?) 
        Dim (2) is the z (However, I do not see a look - eye operation but only look)
        Dim (1) is the up vector, which represents the y axis.
        
        LLFF definition: https://github.com/Fyusion/LLFF
        Dim 5: [image height, image width, focal length]
        Dim 4: homogeneuous coordinate [0, 0, 1]
        Dim 3: z
        Dim 2: x
        Dim 1: -y
    """

    hwf = poses[0, :3, -1:]  # (3, 1)
    center = poses[:, :3, 3].mean(0)  # (3,)
    vec2 = normalize(poses[:, :3, 2].sum(0)) # (3,)
    up = poses[:, :3, 1].sum(0)  # (3,)
    c2w = np.concatenate([viewmatrix(vec2, up, center), hwf], 1)
    
    return c2w