amazon-sumerian-hosts icon indicating copy to clipboard operation
amazon-sumerian-hosts copied to clipboard

Published 20 hours ago • verified publisher icon aws-samples

Mouth Does Not Move With Speech

Open GeneUNCG opened this issue 2 years ago • 7 comments

Hello all, the code has been working magnificently for the past 6 months, but until recently, there is an issue where the mouth of the avatar does not move with the speech of the avatar, the mouth stays shut. I was curious if anyone was able to replicate this issue as this happened after I updated Chrome. I did not make any code changes before this started happening which is interesting.

GeneUNCG avatar Feb 02 '23 02:02 GeneUNCG

I just tried running the demos by grabbing the pulling down the latest code from this repo and was able to successfully run both the Babylon JS demos (npm run start-babylon) and Three.JS demos (npm run start-three) without any issues. I'm using the latest version of Chrome on Mac (Version 109.0.5414.119). This may be an indication that it's an issue specific to your codebase. Do the included demos run correctly for you?

Krxtopher avatar Feb 02 '23 04:02 Krxtopher

I just tried running the demos by grabbing the pulling down the latest code from this repo and was able to successfully run both the Babylon JS demos (npm run start-babylon) and Three.JS demos (npm run start-three) without any issues. I'm using the latest version of Chrome on Mac (Version 109.0.5414.119). This may be an indication that it's an issue specific to your codebase. Do the included demos run correctly for you?

Hey @Krxtopher, I just cloned a fresh copy of the repo and the problem still exists. Are you familiar with this issue? This indicates there is something wrong with one or more environment setting.

GeneUNCG avatar Feb 15 '23 15:02 GeneUNCG

Some questions for you...

  1. Which demo are you running in which you see this problem - is it the Three.js demo or one of the Babylon.js demos?
  2. Which browser(s) and OS are you seeing this issue in?
  3. If you open the JavaScript console in your browser, are there any errors or warnings shown? If so, post them here.
  4. Are you able to create a screen recording showing the issue, including showing how you are launching the demos?

Krxtopher avatar Feb 15 '23 17:02 Krxtopher

Some questions for you...

  1. Which demo are you running in which you see this problem - is it the Three.js demo or one of the Babylon.js demos?
  2. Which browser(s) and OS are you seeing this issue in?
  3. If you open the JavaScript console in your browser, are there any errors or warnings shown? If so, post them here.
  4. Are you able to create a screen recording showing the issue, including showing how you are launching the demos?
  1. I am running all of the Babylon.js demos, each have the issue.
  2. I have used ChromeOS, Edge, and Firefox browsers.
  3. Running the helloWorldDemo for an example, there are no errors, but there is one warning. Here is a screengrab of the entire log: image
  4. Here are screengrabs of how I am launching and using the demo:

Step 1: Run npm run start-babylon in the amazon-sumerian-hosts directory. image

Step 2: Wait for the testing environment to compile and load successfully. image

Step 3: Load the testing environment in the browser (here I am using helloWorldDemo in ChromeOS). image

When clicking the 'Speak' button seen in the above screengrab, the audio plays but no mouth movement occurs.

GeneUNCG avatar Feb 15 '23 23:02 GeneUNCG

Thanks. That extra info gave me enough clues to reproduce the issue on my side. Looks like you've found a previously undiscovered incompatibility between Sumerian Hosts and BabylonJS v5. At the moment, it looks like Hosts will only work with BabylonJS 4.

Is BabylonJS 5 compatibility a requirement for your current project?

I'll submit a temporary fix to the project configuration soon so that the demos are built using BabylonJS 4. That will at least prevent others from running into the issue you did when running the demos. But adding compatibility for BabylonJS 5 will require some time. If you can share with us how important BabylonJS 5 compatibility is that will help us in our roadmap planning.

Krxtopher avatar Feb 16 '23 17:02 Krxtopher

I've created a detailed bug ticket describing the larger BJS v5 compatibility problem. You can use that ticket to track the progress of this bug. "Speech animation, gaze tracking, face shape break with BabylonJS v5"

Krxtopher avatar Feb 16 '23 21:02 Krxtopher

Duplicate of #155 (we can close others and track a single issue, I'm currently investigating to understand what the problem is)

trusktr avatar Feb 17 '24 06:02 trusktr