investigate why depth loss has such a big performance impact on 3D anim mode

[dmarx]

presumably we're already computing depth for the 3D warps: we should get the loss for free. I suspect we are unnecessarily re-estimating depth here

[dmarx]

is the flow being applied to the depth field? that might explain it. and if it's not, it should be.

maybe this is a use case for one of those representations that carries depth information in addition to visual?

[dmarx]

when we sample to backfill occluded pixels (which also... should we maybe just optimize these directly? is sampling just for the init?), are we taking the depth map into consideration? I'm guessing we're not, which probably causes a lot of the weird shape pathologies, where it'll look like a face or something, but then it'll suddenly flatten out onto one edge and smear into like a face generator (like a cellular automata slider generator).