How to use in the FLAME model

[cnpgs]

In the README, it is stated that "We also make a version of our model available in the topology of the FLAME model" (https://github.com/waps101/AlbedoMM#flame-topology-model); I have attempted to use the model in FLAME (by replacing FLAME_texture.npz), but it seems that the number of principal components is different and yields an error that reshaping cannot be done. Thus, is it possible to provide more details on how the albedoModel2020_FLAME_albedoPart.npz model can be used within the FLAME model please?

[TimoBolkart]

Can you please provide some more details about where you replaced the FLAME_texture.npz? We do have some demo code online to sample textures from FLAME_texture or the albedoMM_texture. Does that help you?

[cnpgs]

Many thanks for your speedy response, and apologies for the late reply; I was using a different repo, the PyTorch-based Photometric FLAME Fitting, and attempted to replace the texture model here.

I will check out the demo code that you have mentioned, and see if I can resolve the issue (either by switching to the TensorFlow version, or by trying to adapt the code to enable the use of AlbedoMM in the PyTorch version)...I will post another reply when I have any updates. In the meantime, thanks again for your assistance!

[cnpgs]

I seem to have managed to implement AlbedoMM in PyTorch (for Photometric FLAME fitting and for DECA, which is based on the former repo), and have also executed the demo code; an example of a randomly sampled texture using the demo code is as follows: To my eyes, it seems that the colour is a-bit 'off'/less realistic/paler and also noticed less variations in the generated samples, especially compared to other texture/color spaces such as the MPI texture space:

Am I missing something, or is this simply a characteristic of the model?

[BernhardEgger]

Hi, that looks correct to me. The albedo model provides specular and diffuse albedo maps as described in the paper. The FLAME model (and also the Basel face model) provide a color model that mixes the albedo with some kind of illumination - an as ambient as possible one. One step you might miss in your implementation is to apply gamma (2.2) - that is not done in the FLAME or BFM code but necessary with the albedo model. "The diffuse and specular albedo maps are stored in a nonlinear colour space so we preprocess them by applying inverse gamma (of value 2.2) to transform them back to a linear space" Best Bernhard

[cnpgs]

Hi, that looks correct to me. The albedo model provides specular and diffuse albedo maps as described in the paper. The FLAME model (and also the Basel face model) provide a color model that mixes the albedo with some kind of illumination - an as ambient as possible one.

Thanks very much for your explanations; I was expecting the result of the albedo model to be different to the FLAME and Basel face models, but was still unsure if perhaps I was doing something wrong - so thanks for clarifying.

One step you might miss in your implementation is to apply gamma (2.2) - that is not done in the FLAME or BFM code but necessary with the albedo model. "The diffuse and specular albedo maps are stored in a nonlinear colour space so we preprocess them by applying inverse gamma (of value 2.2) to transform them back to a linear space"

I have applied the inverse gamma on the final rendered texture only, as done in the demo code (seemingly based on equation (11) in the paper); so is there someplace else where it also needs to be applied? (I apologise if this is obvious...as you may tell, I am still quite new to the area :stuck_out_tongue:)

[Bornblack]

I also try to replace texure model in Photometric FLAME fitting as referenc to @TimoBolkart 's demo code，but the optimization process would crash in the non-rigid fitting part as all the loss turn out to be nan.

class FLAMETex(nn.Module):
    """
    current FLAME texture are adapted from BFM Texture Model
    """

    def __init__(self, config):
        super(FLAMETex, self).__init__()
            mu_key = 'MU'
            pc_key = 'PC'
            spec_MU = 'specMU'
            spec_PC = 'specPC'

            tex_path = config.flame_albedo_tex_space_path
            tex_space = np.load(tex_path)
            n_pc = tex_space[pc_key].shape[-1]

            tex_mean = tex_space[mu_key].reshape(1, -1)
            tex_basis = tex_space[pc_key].reshape(-1, n_pc)

            spec_mean = tex_space[spec_MU].reshape(1,-1)
            spec_basis = tex_space[spec_PC].reshape(-1,n_pc)
            
            tex_params = config.tex_params
            tex_mean = torch.from_numpy(tex_mean ).float()[None,...]
            tex_basis = torch.from_numpy(tex_basis [:,:tex_params]).float()[None,...]

            spec_mean = torch.from_numpy(spec_mean).float()[None,...]
            spec_basis = torch.from_numpy(spec_basis[:,:tex_params]).float()[None,...]
          
            self.register_buffer('tex_mean ', tex_mean )
            self.register_buffer('tex_basis ', tex_basis )
            self.register_buffer('spec_mean',spec_mean)
            self.register_buffer('spec_basis',spec_basis)

    def forward(self, texcode):
        diff_albedo = self.tex_mean + (self.tex_basis *texcode[:,None,:]).sum(-1)
        spec_albedo = self.spec_mean + (self.spec_basis*texcode[:,None,:]).sum(-1)
        texture = torch.pow(0.6 * (diff_albedo + spec_albedo), 1.0 / 2.2)
        texture = texture.reshape(texcode.shape[0], 512, 512, 3).permute(0,3,1,2)
        texture = F.interpolate(texture, [256, 256])
        texture = texture[:,[2,1,0], :,:]
        return texture

Any help would be appreciated！

[cnpgs]

@Bornblack : I also had the same issue; I think it's because of the possibility of having negative values when you apply the power (1/2.2); I clipped the minimum to 0, which did not work; I then applied abs(), which also did not work, and finally I applied a small value epsilon which seemed to do the trick (why this is necessary I'm not sure, maybe it's some sort of PyTorch bug...); try it out and see if it solves your issue:

import sys
epsilon = sys.float_info.epsilon

texture = 0.6*(diff_albedo + spec_albedo)
texture = torch.clamp(texture, min=0.0)
texture = torch.pow(texture.abs() + epsilon, 1.0/2.2)

Not sure if it's the correct way/if there's a better solution, and also not sure if both torch.clamp() AND abs() are required (have not yet tested it), but the above code did not yield any NaN values.

[Bornblack]

Aha! It works! It turns out that some values in diff_albedo + spec_albedo are 0, which cause nan during the optimization. The coefficient 0.6 is set just to make diff_albedo + spec_albedo in range [0,1]. The abs() is not required. Thanks! @cnpgs

[cnpgs]

@Bornblack: I see, thanks! Happy to see that it works, glad to be of service :grinning:

[Bornblack]

@TimoBolkart Hi！ Can you explain more about the coefficient 0.6? That is different from what the original paper discribed. I seems wrong to just replace texure model in Photometric FLAME fitting as reference to demo code as the render pass of diffuse albedo map and specular albedo map are different.

[ujjawalcse]

Hey @Bornblack @cnpgs @TimoBolkart @waps101 , How did you get the 3D child model? Did you use DECA or DECA like deep learning model for it. I also tried the DECA, but their texture is of low resolution. What can we do for generating high resolution texture for the Flame model?