uLipSync icon indicating copy to clipboard operation
uLipSync copied to clipboard

Published 20 hours ago • verified publisher icon hecomi

Feature request: Option to calculate delta and the delta of the delta

Open liudger opened this issue 2 years ago • 7 comments

Now the mfcc uses 12 coefficients? Is it possible to calculate also the delta so the coefficients can get to double or triple the amount to have more accurate matching results?

We need to implement Rung Kutta method for this. I am not an expert in programming. Is this doable? Or is the performance hit too high?

https://joinerda.github.io/Solving-ODEs-in-Unity/

What are your thoughts?

liudger avatar Sep 06 '22 11:09 liudger

Sorry for the very late reply.

I would like to explore the option of calculating the DMFCC in multiple frames and adding it behind the current MFCC. In that case, it will have 2N+1 dimensional features, where MFCC is N dimensional and DMCC is N+1 dimensional. Currently, N=12 is fixed, so I will consider adjusting this as well.

hecomi avatar Feb 04 '23 05:02 hecomi

I could try to clean up the code I now have and do a pull request. Not sure when I have time.

liudger avatar Feb 04 '23 13:02 liudger

My previous code for calculating DMFCC was not working correctly. I wanted to start a new try. If you have any draft/ideas to share please let me know.

Meanwhile I found there are options for burst compile to improve speed in cost of precision. Did you test these?

[BurstCompile(FloatMode = FloatMode.Fast)]

and

[BurstCompile(FloatPrecision = FloatPrecision.High)]

liudger avatar May 08 '23 12:05 liudger

Hi @hecomi ,

I think I am quite far along to deploy delta calculation. I will make a draft pull request so you could have a look. I hope to have Tuesday some time to make the pull request and add a demo video. I don't have a solution for the changed profile that needs 24 array length. I had this in my previous code. But it did give some errors (memory leaks) with the different length of profiles.

image

liudger avatar May 28 '23 15:05 liudger

Quick test. Still some memory leaks to tackle

https://github.com/hecomi/uLipSync/assets/4112111/bb6f343d-ba7f-4876-a1b4-b460088024e9

liudger avatar Jun 02 '23 10:06 liudger

every time I create in runtime a new phoneme I get this error:

A Native Collection has not been disposed, resulting in a memory leak. Enable Full StackTraces to get more details.

I shall push the pull request

liudger avatar Jun 02 '23 10:06 liudger

Memory leaks were solved. Forgot to dispose an offset buffer for managing the memory in the buffer with melcepstrum data.

Pull request created #44

liudger avatar Jun 05 '23 07:06 liudger