Questions Regarding the Use of flow_model in Dynamic Scene Modeling

[Rosetta-Leong]

Dear authors,

I am currently working with your codebase and paper on dynamic scene modeling, and I have some questions regarding the implementation and purpose of the FlowHead.

In the context of modeling dynamic scenes, the paper mentions both a FlowHead and a deform_MLP. I am trying to understand the distinction between these two components:

1. What is the difference between the FlowHead and the earlier deform_MLP?

Specifically, why is it necessary to introduce a separate FlowHead in addition to the deform_MLP when modeling dynamic scenes? How do their roles differ in the overall framework?

2. Usage of FlowHead in the code:

I noticed that the flow generated by the FlowHead does not seem to be utilized elsewhere in the code. Could you clarify the purpose of the flow output by the FlowHead? Is it intended for applying constraints, for visualization purposes, or does it serve another function?

I appreciate your time and assistance in addressing these questions. Your insights will greatly help in understanding and utilizing the framework more effectively.

Best regards, Rosetta

[markomih]

Hi @Rosetta-Leong,

Thank you for your interest in our work!

1. What is the difference between the FlowHead and the earlier deform_MLP?

Our primary focus was on reconstructing dynamic sequences from multi-view images, such as those in the Owlii dataset. In these cases, the reconstruction process is almost fully constrained by the photometric rendering loss. The challenge with neural modules is that they often lack sufficient capacity to model long and complex sequences with challenging motions. By using separate FlowHead and deform_MLP modules, we increase the learning capacity, which improves the results for these dynamic sequences. However, in very sparse setups (e.g., only 4 views), the benefit of having two separate deformation/flow modules is reduced, as the added capacity is less critical in such cases.

2. Usage of FlowHead in the code:

The FlowHead module deforms the points (the means3D variable) in the following section of the code: https://github.com/markomih/SplatFields/blob/main/utils/time_utils.py#L503C19-L503C26

Best regards, Marko