eos

eos copied to clipboard

You've mentioned adding a convergence criterion as a todo in fitting::fit_shape_and_pose. I'm interested in exploring this further. What do you suggest as the convergence condition?

I'm currently testing different number of iterations for a single image fitting. It seems that the fitting works best with 3-5 iterations, then it gets worse with 10 iterations. Image: Fitting with 3 iterations: Fitting with 10 iterations:

My observation is that it takes up to 50 or so iterations for the parameters to fully converge. However, visually, there is usually no difference after 5 or so iterations. See for example below graphs for one particular image.

In your example image, it looks like it's overfitting a bit. How does it look after 50 iterations? The same as after 10? And the solution in this case would be to use a bit of a larger regularisation. What are you using for lambda, 30?

I'm a bit hesitant to introduce an automatic convergence criterion for this function, as it would be hard to choose one. If we stop when the parameter change is sufficiently small, it may run up to 50 iterations, which would not work for real-time applications anymore. If the desired output is just an accurate mesh, you can stop after 1-5 iterations, as after that there are no visual changes anymore (so the convergence criterion could e.g. be the landmark-reprojection error). So it really depends on the individual application. I'm very open to suggestions though! :-)

I was using a regularisation of 15 in my previous example. You're right. I tried bumping it up to 100, and it works great on an image of 660x400 :D

I tried using the same image and lambda but with a smaller resolution of 640x360, and the result is less satisfactory. Adjusting lambda to 50, 300, 500 did not work either.

I can't see how lambda is related to the image resolution. Are there other parameters I should adjust depending on the image size? I tried adjusting distance_threshold in fitting::find_occluding_edge_correspondences, but I'm not sure how it should be calculated. The distance_threshold was 180 in my previous two examples.

Hmm, so you were changing the image resolution from 660x400 to 640x360? The uploaded images are of size 400x241 and 400x225, but it may well be that GitHub resizes the images. I am asking because I'm interested in whether you changed the aspect ratio - in second image you uploaded, the aspect ratio is changed, because the image only got rescaled in y-direction. So the face is basically squeezed to an unnatural one, and what may have happened is that the model can't explain this squeezed face very well, which may explain your result. Still, the squeezing in your example seems rather small, and I am surprised that the model can't simply explain that with its shape variation.

Can you double-check what the exact image resolutions are, and then try rescaling the image but keeping the aspect ratio?

The fitting in general is (or should be) independent of image resolution. distance_threshold should be as well - here, it is multiplied with the scale. I'm actually not 100% confident that that formula is right, which is why I added the note, but in my tests it seemed to do the right thing.

Also there's a note about the "units" of distance_threshold in the function documentation above.

@wli75 Did you look at any of the things I mentioned and find out more?

In any case I'll leave this issue open as it would be a pretty nice addition to have an automatic convergence criterion :-)

Apologies for the delay. I'm planning to look into this some time in late-April. I probably got some things mixed up with the resolutions of my test images.