Can we get back the expression parameters from the Flame vertices and faces?

[ujjawalcse]

Hey @Rubikplayer @raffienficiaud @TimoBolkart , Suppose we're predicting flame vertices using any model. Can we get back the pose ,shape and expression parameters or just the expression parameters ?

Thanks.

[TimoBolkart]

Hey, given FLAME vertices (output by any method) you can get all FLAME parameters back (i.e., pose, shape, expression) with a simple optimization. You just need to minimize the difference of the given FLAME vertices and FLAME w.r.t. the parameters. We do have some code available to do this, e.g., TF_FLAME/fit_3D_mesh. You can also easily adapt the flame-fitting/fit_lmk3d in this repository by replacing the landmark error (lmk_err) by the difference of given FLAME vertices and FLAME (i.e., model).

[ujjawalcse]

Thanks @TimoBolkart , Let me try this.

[ujjawalcse]

Hey @TimoBolkart , I tried your method here , https://github.com/TimoBolkart/TF_FLAME/blob/master/fit_3D_mesh.py It's able to produce all the parameters. But it's getting too slow when i'm calling the function in a loop of vertices. Initially, it was taking around 6-7 sec only. But as iteration grows it's taking up to 40-50 sec or even more. Also, the RAM used is gradually increasing from 5-6 GB at start to 32 GB as the iteration grows. Sharing the code for your reference,

def fit_3D_mesh(target_vertices, model_fname, weights, show_fitting=True):
    '''
    Fit FLAME to 3D mesh in correspondence to the FLAME mesh (i.e. same number of vertices, same mesh topology)
    :param target_3d_mesh_fname:    target 3D mesh filename
    :param model_fname:             saved FLAME model
    :param weights:             weights of the individual objective functions
    :return: a mesh with the fitting results
    '''

    #target_mesh = Mesh(filename=target_3d_mesh_fname)

    tf_trans = tf.Variable(np.zeros((1,3)), name="trans", dtype=tf.float64, trainable=True)
    tf_rot = tf.Variable(np.zeros((1,3)), name="pose", dtype=tf.float64, trainable=True)
    tf_pose = tf.Variable(np.zeros((1,12)), name="pose", dtype=tf.float64, trainable=True)
    tf_shape = tf.Variable(np.zeros((1,300)), name="shape", dtype=tf.float64, trainable=True)
    tf_exp = tf.Variable(np.zeros((1,100)), name="expression", dtype=tf.float64, trainable=True)
    t1=time.time()
    smpl = SMPL(model_fname)

    tf_model = tf.squeeze(smpl(tf_trans,
                               tf.concat((tf_shape, tf_exp), axis=-1),
                               tf.concat((tf_rot, tf_pose), axis=-1)))
    t2=time.time()
    print('Time taken in SMPL part:',t2-t1)
    #print('tf_exp before fitting:',tf_exp)
    with tf.Session() as session:
        session.run(tf.global_variables_initializer())
        t3=time.time()
        mesh_dist = tf.reduce_sum(tf.square(tf.subtract(tf_model, target_vertices)))
        neck_pose_reg = tf.reduce_sum(tf.square(tf_pose[:,:3]))
        jaw_pose_reg = tf.reduce_sum(tf.square(tf_pose[:,3:6]))
        eyeballs_pose_reg = tf.reduce_sum(tf.square(tf_pose[:,6:]))
        shape_reg = tf.reduce_sum(tf.square(tf_shape))
        exp_reg = tf.reduce_sum(tf.square(tf_exp))
        t4=time.time()
        print('Time in reg:',t4-t3)
        # Optimize global transformation first
        vars = [tf_trans, tf_rot]
        loss = mesh_dist
        t5=time.time()
        optimizer = scipy_pt(loss=loss, var_list=vars, method='BFGS', options={'disp': 1})
        print('Optimize rigid transformation')
        optimizer.minimize(session)
        t6=time.time()
        print('Time in 1st optimization :',t6-t5)
        # Optimize for the model parameters
        vars = [tf_trans, tf_rot, tf_pose, tf_shape, tf_exp]
        loss = weights['data'] * mesh_dist + weights['shape'] * shape_reg + weights['expr'] * exp_reg + \
               weights['neck_pose'] * neck_pose_reg + weights['jaw_pose'] * jaw_pose_reg + weights['eyeballs_pose'] * eyeballs_pose_reg
        t7=time.time()
        optimizer = scipy_pt(loss=loss, var_list=vars, method='BFGS', options={'disp': 1})
        print('Optimize model parameters')
        optimizer.minimize(session)
        t8=time.time()
        print('Time in 2nd optimization :',t8-t7)
        print('Fitting done')

        # if show_fitting:
        #     # Visualize fitting
        #     mv = MeshViewer()
        #     fitting_mesh = Mesh(session.run(tf_model), smpl.f)
        #     fitting_mesh.set_vertex_colors('light sky blue')

        #     mv.set_static_meshes([target_mesh, fitting_mesh])
        #     six.moves.input('Press key to continue')

        #return Mesh(session.run(tf_model), smpl.f)
        #print('tf_exp after fitting:',session.run(tf_exp))
        return session.run(tf_exp)[0],session.run(tf_shape)[0],session.run(tf_pose)[0]
        session.close()
        del session

And, I'm calling this function like this,

for i in range(num):
        target_vertices=predicted_vertices[i]
        exp,shape,pose=fit_3D_mesh(target_vertices, flame_model_path, weights, show_fitting=show_fitting)

I also tested with adding session.close() and del session, but no luck. Then, printed timing for some part, I found the both optimizers (at line 65 and 74) time and space (RAM used) are gradually increasing with the iterations Please suggest some way out. Thanks.