Sound Entities

[HifiExperiments]

Closes #69

Introduces a new Entity Type, Sound, which provides the same functionality as the Sound scripting API but in a more accessible way. They have the following properties:

 * @property {string} soundURL="" - The URL of the sound to play, as a wav, mp3, or raw file.  Supports stereo and ambisonic.
 * @property {boolean} playing=true - Whether or not the sound should play.
 * @property {number} volume=1.0 - The volume of the sound, from <code>0</code> to <code>1</code>.
 * @property {number} pitch=1.0 - The relative sample rate at which to resample the sound, within +/- 2 octaves.
 * @property {number} timeOffset=0.0 - The time (in seconds) at which to start playback within the sound file.  If looping,
 *     this only affects the first loop.
 * @property {boolean} loop=true - Whether or not to loop the sound.
 * @property {boolean} positional=true - Whether or not the volume of the sound should decay with distance.
 * @property {boolean} localOnly=false - Whether or not the sound should play locally for everyone (unsynced), or synchronously
 *     for everyone via the Entity Mixer.

Also introduces a new Entities function, Entities.restartSound, which can restart a localOnly sound entity.

Thank you again to @AleziaKurdis for the icons!

Notes

• timeOffset only affects the first loop. this is the existing behavior for the Sound API also but I don't know if it's desirable?
• when localOnly is false, the sounds notably play through the Entity mixer, not the Audio mixer (or rather, are sent from the Entity mixer to the Audio mixer to be mixed for everyone). This is because, surprisingly, injectors on the Audio mixer don't work and I'm not sure why! If we want to investigate this in the future, we can do so without any protocol changes, so it isn't a big deal.

Funding

This project is funded through NGI0 Entrust, a fund established by NLnet with financial support from the European Commission's Next Generation Internet program. Learn more at the NLnet project page.

[vegaslon]

The sound is playing slower then normal on default playing of a pitch of 1, if I change the pitch to 2 it appears to play closer to the right right speed. I used this sound file https://files.freemusicarchive.org/storage-freemusicarchive-org/music/no_curator/Scott_Holmes/Inspiring__Upbeat_Music/Scott_Holmes_-04-_Upbeat_Party.mp3

Here is a json that will rez a jukebox that pretty much plays the sound on the entity mixer on default settings besides a lower volume. You just click the cube to turn on sound and click the cube again to turn it off. Server Side JukeBox example.json

the scripts for that are on https://glitch.com/edit/#!/server-jukebox?path=musicplayerclickserver.js%3A18%3A195 if you want to remix them and mess around with anything yourself.

[HifiExperiments]

@vegaslon interesting! you're saying that's an existing bug or new to this PR? I see you're using the Sound API in those scripts, so I assume the same issue will affect these new Sound entities, since they are the same internally. I didn't really touch the internal behavior of sounds, and I don't think I experienced that in my testing. I wonder if that's a stereo mp3? we have some logic that multiplies things by 2 for stereo files, and since sounds have been hard to make so far, I could definitely imagine there being bugs. but it might be best to file a separate issue for it

[vegaslon]

New to this PR the sound injectors play at normal speed while your sound entities are playing slower.

[HifiExperiments]

@vegaslon oooooooooo fascinating. I'll check it out

[SilverfishVR]

Very weird, I can sort of confirm that the provided test clip plays too slow, but also, setting it to 2 makes the speed correct but it does not sound right, it sounds like it has been mangled by pitch bending or time stretching somehow. playing it as pitch 2 on a soundEntity sounds significantly different to how it sounds played in my browser or by a script at 1 pitch

With my own files I don't seem to have this issue. The Sound entity plays at the same speed and pitch as a script 🤔

cube with serverEntityScript to play sound: http://silverfish-freestuff.s3.amazonaws.com/TestStuff/SoundEntities/testinjector.json use Entities.callEntityServerMethod([UUID], "playSound"); to make it play.

the sound: http://silverfish-freestuff.s3.amazonaws.com/TestStuff/SoundEntities/dtmf2.mp3

Weird thing is, in my mind "pitch" used to be just that, adjust the pitch, but not the playback speed. Why else would it be called pitch and not speed? Also, the API docs lists the range as being 0.0625 to 16 corresponding to ±2 octaves, but to my (untrained ear), setting it to 2 doubles both the speed and frequency (1 octave).

But I could be remembering wrongly, because on current release it is also just a speed adjustment.

[HifiExperiments]

thanks folks - it turns out the issue related to stereo sound files. they should work now!

@SilverfishVR our pitch adjustments have indeed always also affected speed. this is because we adjust the pitch by simply changing the sampling speed. this is a common simple technique for changing pitch (https://en.wikipedia.org/wiki/Pitch_control), but it would be awesome if in the future we could support other techniques (https://en.wikipedia.org/wiki/Audio_time_stretching_and_pitch_scaling)

[vegaslon]

Just to confirm, with these audio entities there is no way to play a sound globally from a client? That would still exclusively be a feature of scripted audio injectors?

[HifiExperiments]

if you set localOnly to false, the sound will play globally for everyone in that domain, is that what you mean?

I guess one subtle difference between this and existing Audio injectors is the ability to play a sound globally and have it follow you from domain to domain without losing its place in the sound. with Sound entities, if you have an avatar entity with localOnly = false, the sound will restart when you enter a new domain. if you have an audio injector playing with localOnly = false, it will keep its place in the clip when you switch domains, since it's playing directly from your client. is that an acceptable trade off?

(personally I think yes, since that seems like a rare use case, and you can still accomplish it with the old audio injectors, which aren't going anywhere.)

[vegaslon]

Yes I was just making sure did not need to test for it. Does this mean you always have permission to attach a sound to the entity mixer, no matter your create status as long as have permission to have avatar entities?

I think being able to create sound that way is fine since it would always be coupled to your avatar and hence would go away with you, ideally.

[HifiExperiments]

oooooo actually you're totally right, avatar entities shouldn't go to the entity mixer so avatar entity sound entities might not be working properly right now, I will look into that. they'll need to play through the avatar mixer...or even better directly through the audio mixer like they're supposed to

[vegaslon]

Besides the avatar entity question, I see no functionality problems with this pr.

[HifiExperiments]

this was way simpler than I was making it. non-localOnly avatar entities can just be played as non-localOnly injectors by their owning avatar's client, just like normal audio injectors. so if you make an avatar entity with localOnly = false, it should be able to follow you from domain to domain (without restarting)

[HifiExperiments]

@1st-BrainStormer noticed I wasn't properly handling pivot/registration point, which can be used for some interesting effects with spatialized sounds. I've added that and fixed some deadlocks that it revealed. should be ready to merge once that's tested!

[HifiExperiments]

talking with @1st-BrainStormer, all the deadlock issues are fixed, but it seems non-local sounds don't update their transforms on the entity server as a result of physics, which sounds like https://github.com/overte-org/overte/issues/926

although...shouldn't mixers still get updates in the form of entity edits as clients simulate the physics? I need to investigate some more.

[HifiExperiments]

I believe I've fixed that last issue! if you make a localOnly = false, positional = true sound and make it the child of a rotating box, the sound will now properly spatialize.

there's maybe still one remaining issue where those sounds sometimes glitch out for a frame, so I will investigate that

[HifiExperiments]

hm actually I can't reproduce that anymore, I might have had my server set up weirdly yesterday or something...

not sure if we want to QA or CR again, but I believe this is ready for merge now. I'm assuming we will do another round of testing once protocol_changes is merged to master anyways or for the RC

[ksuprynowicz]

I also think it's safe to merge now