Teleporting lips when person in the video is not speaking

[batiray]

Videos that include constant talking audio and shots of person speaking and person not speaking have a problem. When person in the video is not speaking, lip is trying to be synced and the person's lips are teleported from one place to other. How can we implement a condition that if person's lips are not moving, lips are not tried to be synced?

Many thanks.