vircadia-native-core icon indicating copy to clipboard operation
vircadia-native-core copied to clipboard

Published 20 hours ago • verified publisher icon vircadia

Optimization coding pass over AudioMixerSlavePool and AvatarMixerSlavePool pt. 2.

Open odysseus654 opened this issue 3 years ago • 51 comments

This is an optimization pass over the mutex logic in AudioMixerSlavePool and AvatarMixerSlavePool (which is effectively the same code).

This is a continuation of the changes in #1358 , with more of a focus on contracts and invariants and effectively no use of mutexes. This should (almost by definition) have reduced lock contention, but "thread-safety through the use of code comments" may have negative side effects. I don't know which of these PRs will end up being used (perhaps a mixture of both?)

Additional changes made to this PR:

  • Removed all use of "friend" by factoring out shared variables into a private class that the slaves can see but pool users cannot
  • A number of counters have been changed from "int" to "std::atomic"
  • Much documentation of invariants and code contracts

Testing is almost identical to #1358 :

  • Smoketesting: make sure that audio and avatars still work on a server built with this (requires a server with multiple users)
  • Race conditions: exercise a server long enough to know whether there are any lockups or crashes that occur in these two areas
  • Profile: see if this improves the profiling numbers that have been described in issue High CPU usage of assignment client when idle on Linux #1355

odysseus654 avatar Sep 24 '21 04:09 odysseus654

Hey, if anyone wants to use std::shared_mutex, there's a polyfill in the commits (that was later yanked out) ;-)

odysseus654 avatar Sep 24 '21 05:09 odysseus654

The following links are available: build (macOS-latest, client)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-Interface-PR1361-ba09d2a-_536b1ed1.dmg

build (macOS-latest, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-ba09d2a-_536b1ed1.dmg

build (ubuntu-18.04, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-ba09d2a-_536b1ed1.sh
  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-ba09d2a-.tar_536b1ed1.Z
  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-ba09d2a-.tar_536b1ed1.gz

build (windows-latest, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-ba09d2a-_536b1ed1.exe

vircadia-build-notifier avatar Sep 24 '21 06:09 vircadia-build-notifier

Made suggested changes to this PR:

  • changed "slave" to "worker" when dealing with classes in this file
  • moved the "data" struct to be a private member instead of a base class

Attempted to factor out common code to a template base class but... can't do that and still use Qt

odysseus654 avatar Sep 24 '21 07:09 odysseus654

The following links are available: build (ubuntu-18.04, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-2dff102-.tar_aa2c3074.gz
  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-2dff102-_aa2c3074.sh
  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-2dff102-.tar_aa2c3074.Z

build (macOS-latest, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-2dff102-_aa2c3074.dmg

build (windows-latest, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-2dff102-_aa2c3074.exe

build (macOS-latest, client)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-Interface-PR1361-2dff102-_aa2c3074.dmg

vircadia-build-notifier avatar Sep 24 '21 09:09 vircadia-build-notifier

The following links are available: build (ubuntu-18.04, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-5c42466-.tar_569afb18.gz
  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-5c42466-.tar_569afb18.Z
  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-5c42466-_569afb18.sh

build (macOS-latest, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-5c42466-_569afb18.dmg

build (windows-latest, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-5c42466-_569afb18.exe

build (macOS-latest, client)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-Interface-PR1361-5c42466-_569afb18.dmg

vircadia-build-notifier avatar Sep 24 '21 19:09 vircadia-build-notifier

Added commit: renamed _pool to _data as suggested.

odysseus654 avatar Sep 25 '21 03:09 odysseus654

The following links are available: build (macOS-latest, client)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-Interface-PR1361-3dd0fb4-_bc5d9de4.dmg

build (macOS-latest, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-3dd0fb4-_bc5d9de4.dmg

build (ubuntu-18.04, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-3dd0fb4-.tar_bc5d9de4.Z
  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-3dd0fb4-_bc5d9de4.sh
  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-3dd0fb4-.tar_bc5d9de4.gz

build (windows-latest, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-3dd0fb4-_bc5d9de4.exe

vircadia-build-notifier avatar Sep 25 '21 04:09 vircadia-build-notifier

Linux build of the current version fails for me with following error:

/home/ksuprynowicz/Vircadia/source/assignment-client/src/audio/AudioMixerSlavePool.cpp:216:37: error: ‘_slaves’ was not declared in this scope; did you mean ‘_xsaves’?
  216 |     assert(_data.numThreads == (int)_slaves.size());

ksuprynowicz avatar Sep 25 '21 14:09 ksuprynowicz

Gack, was worried something like that might happen. I haven't been using debugging builds so assert() is meaningless to me. I'll fix that.

odysseus654 avatar Sep 25 '21 17:09 odysseus654

Might be worth trying to test this with #1149

daleglass avatar Sep 25 '21 18:09 daleglass

The following links are available: build (macOS-latest, client)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-Interface-PR1361-12dc362-_45d33127.dmg

build (macOS-latest, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-12dc362-_45d33127.dmg

build (ubuntu-18.04, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-12dc362-.tar_45d33127.Z
  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-12dc362-_45d33127.sh
  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-12dc362-.tar_45d33127.gz

build (windows-latest, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-12dc362-_45d33127.exe

vircadia-build-notifier avatar Sep 25 '21 19:09 vircadia-build-notifier

Current version builds well but assignment client child threads continuously crash with message:

assignment-client: /home/ksuprynowicz/Vircadia/source/assignment-client/src/audio/AudioMixerSlavePool.cpp:44: AudioMixerWorkerThread::wait()::<lambda()>: Assertion `_data.numStarted <= _data.numThreads' failed.
assignment-client: /home/ksuprynowicz/Vircadia/source/assignment-client/src/audio/AudioMixerSlavePool.cpp:58: void AudioMixerWorkerThread::notify(bool): Assertion `_data.numFinished < _data.numThreads && _data.numFinished <= _data.numStarted' failed.
assignment-client: /home/ksuprynowicz/Vircadia/source/assignment-client/src/audio/AudioMixerSlavePool.cpp:58: void AudioMixerWorkerThread::notify(bool): Assertion `_data.numFinished < _data.numThreads && _data.numFinished <= _data.numStarted' failed.
[09/25 23:33:28] [CRITICAL] [default] [26583] [assignment-client-monitor] Child process 27216 on port 0has crashed with exit code 6. Removed from internal map.

ksuprynowicz avatar Sep 25 '21 21:09 ksuprynowicz

I read this report as "asserts are good" and "next time smoketest the darned thing". Will look, hopefully a quick fix.

odysseus654 avatar Sep 25 '21 22:09 odysseus654

I'm unable to smoketest this on my windows machine, so made changes in the code based directly on your printed error messages.

odysseus654 avatar Sep 26 '21 00:09 odysseus654

The following links are available: build (macOS-latest, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-5686023-_3a4e8a7f.dmg

build (macOS-latest, client)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-Interface-PR1361-5686023-_3a4e8a7f.dmg

build (ubuntu-18.04, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-5686023-_3a4e8a7f.sh
  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-5686023-.tar_3a4e8a7f.Z
  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-5686023-.tar_3a4e8a7f.gz

build (windows-latest, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-5686023-_3a4e8a7f.exe

vircadia-build-notifier avatar Sep 26 '21 02:09 vircadia-build-notifier

The current version crashes with a different message. From domain server side:

[09/26 09:46:15] [DEBUG] [hifi.domain_server] Haven't heard from  213.134.187.199:24938 ""  in  11895  msec
[09/26 09:46:16] [DEBUG] [hifi.networking.ice] Removing silent node "Avatar Mixer" (W) {0c89c76d-aa3c-4a02-bc92-543f4e5bbb91}(53345) 213.134.187.199:24938 / 192.168.0.15:56776

and from assignment client side:

[09/26 09:46:03] [WARNING] [default] [54542] [avatar-mixer] Setting::Interface::init() for key "private/selectedMetaverseURL" - Manager not yet created. Settings persistence disabled.
assignment-client: /home/ksuprynowicz/Vircadia/source/assignment-client/src/avatars/AvatarMixerSlavePool.cpp:54: void AvatarMixerWorkerThread::notify(bool): Assertion `_data.numFinished < _data.numThreads' failed.
assignment-client: /home/ksuprynowicz/Vircadia/source/assignment-client/src/avatars/AvatarMixerSlavePool.cpp:54: void AvatarMixerWorkerThread::notify(bool): Assertion `_data.numFinished < _data.numThreads' failed.
[09/26 09:46:03] [CRITICAL] [default] [54378] [assignment-client-monitor] Child process 54542 on port 0has crashed with exit code 6. Removed from internal map.
[09/26 09:46:04] [DEBUG] [hifi.networking] [54383] [entity-server] LimitedNodeList::sendPacket called without active socket for node "Avatar Mixer" (W) {0c89c76d-aa3c-4a02-bc92-543f4e5bbb91}(53345) 213.134.187.199:24938 / 192.168.0.15:56776 - not sending
[09/26 09:46:04] [DEBUG] [hifi.networking] [54385] [entity-script-server] LimitedNodeList::sendPacket called without active socket for node "Avatar Mixer" (W) {0c89c76d-aa3c-4a02-bc92-543f4e5bbb91}(53345) 213.134.187.199:24938 / 192.168.0.15:56776 - not sending

ksuprynowicz avatar Sep 26 '21 07:09 ksuprynowicz

It seems that audio thread crashes too, with same message as last time:

assignment-client: /home/ksuprynowicz/Vircadia/source/assignment-client/src/audio/AudioMixerSlavePool.cpp:58: void AudioMixerWorkerThread::notify(bool): Assertion `_data.numFinished < _data.numThreads' failed.
assignment-client: /home/ksuprynowicz/Vircadia/source/assignment-client/src/audio/AudioMixerSlavePool.cpp:58: void AudioMixerWorkerThread::notify(bool): Assertion `_data.numFinished < _data.numThreads' failed.
[09/26 09:52:02] [CRITICAL] [default] [54699] [assignment-client-monitor] Child process 54867 on port 0has crashed with exit code 6. Removed from internal map.

ksuprynowicz avatar Sep 26 '21 07:09 ksuprynowicz

Thank you for your efforts on this. I can't guarantee that this won't be the last error (likely will be at least a few more, considering the lack of smoketesting)

I've made another commit to ensure that newly created workers are properly waiting (and not active) before leaving "resize"

odysseus654 avatar Sep 26 '21 18:09 odysseus654

The following links are available: build (macOS-latest, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-2611422-_83f2b526.dmg

build (ubuntu-18.04, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-2611422-_83f2b526.sh
  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-2611422-.tar_83f2b526.Z
  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-2611422-.tar_83f2b526.gz

build (macOS-latest, client)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-Interface-PR1361-2611422-_83f2b526.dmg

build (windows-latest, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-2611422-_83f2b526.exe

vircadia-build-notifier avatar Sep 26 '21 20:09 vircadia-build-notifier

I just gave a try for the most recent build, and it crashes in a different way now. Audio mixer crashes this way:

[09/27 20:55:39] [DEBUG] [hifi.assignment-client] [111780] [assignment-client] Received a PacketType::CreateAssignment - attempting to unpack.
[09/27 20:55:39] [DEBUG] [default] [111780] [<audio-mixer>] resize: set 40 threads (was 0)
assignment-client: /home/ksuprynowicz/Vircadia/source/assignment-client/src/audio/AudioMixerSlavePool.cpp:56: void AudioMixerWorkerThread::wait(): Assertion `_function' failed.
assignment-client: /home/ksuprynowicz/Vircadia/source/assignment-client/src/audio/AudioMixerSlavePool.cpp:56: void AudioMixerWorkerThread::wait(): Assertion `_function' failed.
assignment-client: /home/ksuprynowicz/Vircadia/source/assignment-client/src/audio/AudioMixerSlavePool.cpp:56: void AudioMixerWorkerThread::wait(): Assertion `_function' failed.
[09/27 20:55:39] [CRITICAL] [default] [111590] [assignment-client-monitor] Child process 111780 on port 0has crashed with exit code 6. Removed from internal map.

And avatar mixer crashes this way:

[09/27 20:57:09] [DEBUG] [hifi.assignment-client] [111884] [assignment-client] Received a PacketType::CreateAssignment - attempting to unpack.
[09/27 20:57:09] [DEBUG] [default] [111884] [<avatar-mixer>] resize: set 40 threads (was 0)
assignment-client: /home/ksuprynowicz/Vircadia/source/assignment-client/src/avatars/AvatarMixerSlavePool.cpp:52: void AvatarMixerWorkerThread::wait(): Assertion `_function' failed.
assignment-client: /home/ksuprynowicz/Vircadia/source/assignment-client/src/avatars/AvatarMixerSlavePool.cpp:52: void AvatarMixerWorkerThread::wait(): Assertion `_function' failed.
[09/27 20:57:09] [CRITICAL] [default] [111590] [assignment-client-monitor] Child process 111884 on port 0has crashed with exit code 6. Removed from internal map.

ksuprynowicz avatar Sep 27 '21 18:09 ksuprynowicz

Thank you for your efforts on this. I can't guarantee that this won't be the last error (likely will be at least a few more, considering the lack of smoketesting)

I'll be happy to test it as many times as needed :) Thank you for the work you are putting into it.

ksuprynowicz avatar Sep 27 '21 18:09 ksuprynowicz

Heh, plus those asserts you got were just added yesterday; was suspecting this might be the underlying causes of the crashes (rather than just "misbehaviors").

odysseus654 avatar Sep 27 '21 19:09 odysseus654

Okay did some code re-ordering w.r.t. worker thread startup, assuming that these crashes are the first time through the loop.

Less likely chance of this crash but no-crash doesn't mean it works; I'm really wanting to not see any assert()s in this.

odysseus654 avatar Sep 27 '21 19:09 odysseus654

The following links are available: build (ubuntu-18.04, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-fd90b96-.tar_df59a42c.gz
  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-fd90b96-_df59a42c.sh
  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-fd90b96-.tar_df59a42c.Z

build (macOS-latest, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-fd90b96-_df59a42c.dmg

build (macOS-latest, client)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-Interface-PR1361-fd90b96-_df59a42c.dmg

build (windows-latest, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-fd90b96-_df59a42c.exe

vircadia-build-notifier avatar Sep 27 '21 21:09 vircadia-build-notifier

It crashes with both assertions now:

[09/28 09:11:59] [DEBUG] [default] [36663] [<avatar-mixer>] resize: set 40 threads (was 0)
assignment-client: /home/ksuprynowicz/Vircadia/source/assignment-client/src/avatars/AvatarMixerSlavePool.cpp:59: void AvatarMixerWorkerThread::wait(): Assertion `_function' failed.
assignment-client: /home/ksuprynowicz/Vircadia/source/assignment-client/src/avatars/AvatarMixerSlavePool.cpp:59: void AvatarMixerWorkerThread::wait(): Assertion `_function' failed.
assignment-client: /home/ksuprynowicz/Vircadia/source/assignment-client/src/avatars/AvatarMixerSlavePool.cpp:18: virtual void AvatarMixerWorkerThread::run(): Assertion `_data.numStarted < _data.numThreads' failed.
[09/28 09:11:59] [CRITICAL] [default] [36202] [assignment-client-monitor] Child process 36663 on port 0has crashed with exit code 6. Removed from internal map.

[09/28 09:13:29] [DEBUG] [default] [36724] [<audio-mixer>] resize: set 40 threads (was 0)
assignment-client: /home/ksuprynowicz/Vircadia/source/assignment-client/src/audio/AudioMixerSlavePool.cpp:63: void AudioMixerWorkerThread::wait(): Assertion `_function' failed.
[09/28 09:13:29] [CRITICAL] [default] [36202] [assignment-client-monitor] Child process 36724 on port 0has crashed with exit code 6. Removed from internal map.

ksuprynowicz avatar Sep 28 '21 07:09 ksuprynowicz

okay dug into "exit code 6" and it looks like this crash basically happens the moment an assert() failure occurs.

Still looks like we're having problems with new worker startup (I think), I'll walk through that chain of logic a bit further.

odysseus654 avatar Sep 28 '21 07:09 odysseus654

Okay new changes (as described in the commit):

  • New threads now unconditionally enter a wait without checking if numStarted==numThreads (as numThreads is changing out-of-band in this case. Yes it's violating the contract in the code comments, the worker thread is "active" without "owning" the state. Hopefully not too many more headaches with this)
  • Moved numStarted++ inside the workerLock so as to avoid spurious worker thread wakes

odysseus654 avatar Sep 28 '21 07:09 odysseus654

The following links are available: build (macOS-latest, client)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-Interface-PR1361-56fe8f8-_5389d3fd.dmg

build (macOS-latest, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-56fe8f8-_5389d3fd.dmg

build (ubuntu-18.04, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-56fe8f8-.tar_5389d3fd.Z
  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-56fe8f8-.tar_5389d3fd.gz
  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-56fe8f8-_5389d3fd.sh

build (windows-latest, full)

  • https://athena-public.s3.eu-west-3.amazonaws.com/builds/PR/vircadia_vircadia/PR1361/Vircadia-PR1361-56fe8f8-_5389d3fd.exe

vircadia-build-notifier avatar Sep 28 '21 09:09 vircadia-build-notifier

Right now it crashes with a message:

[09/28 16:29:52] [DEBUG] [hifi.assignment-client] [113656] [assignment-client] Received a PacketType::CreateAssignment - attempting to unpack.
[09/28 16:29:52] [DEBUG] [default] [113656] [<audio-mixer>] resize: set 40 threads (was 0)
assignment-client: /home/ksuprynowicz/Vircadia/source/assignment-client/src/audio/AudioMixerSlavePool.cpp:65: void AudioMixerWorkerThread::wait(bool): Assertion `_function' failed.
assignment-client: /home/ksuprynowicz/Vircadia/source/assignment-client/src/audio/AudioMixerSlavePool.cpp:65: void AudioMixerWorkerThread::wait(bool): Assertion `_function' failed.
[09/28 16:29:52] [CRITICAL] [default] [113574] [assignment-client-monitor] Child process 113656 on port 0has crashed with exit code 6. Removed from internal map.

and

[09/28 16:31:22] [DEBUG] [hifi.assignment-client] [113725] [assignment-client] Received a PacketType::CreateAssignment - attempting to unpack.
[09/28 16:31:22] [DEBUG] [default] [113725] [<avatar-mixer>] resize: set 40 threads (was 0)
assignment-client: /home/ksuprynowicz/Vircadia/source/assignment-client/src/avatars/AvatarMixerSlavePool.cpp:61: void AvatarMixerWorkerThread::wait(bool): Assertion `_function' failed.
[09/28 16:31:22] [CRITICAL] [default] [113574] [assignment-client-monitor] Child process 113725 on port 0has crashed with exit code 6. Removed from internal map.

ksuprynowicz avatar Sep 28 '21 14:09 ksuprynowicz

Starting to think I may not know how condition_var.wait() works. Adding a mutex blocker to prevent the wait from occurring before all the threads have started up.

odysseus654 avatar Sep 28 '21 16:09 odysseus654