main thread re-enter without Asyncify

[jiulongw]

I understand that when Asyncify is enabled, main thread can be re-entered during blocking calls. But I find that even when Asyncify is not enabled, re-enter still happens. This is a problem because in all other platforms, worker threads posting to main thread is not expected to run until main thread enters message loop. In our case, re-enter can happen when main thread is in a scoped_lock, causing deadlock. Simple work around is to replace mutex with recursive_mutex, but it adds unnecessary cost to other platforms.

Here's a simple program that shows the problem. You can play with it: https://github.com/jiulongw/emscripten-mutex-test

#include <iostream>
#include <mutex>
#include <thread>

#include <emscripten/threading.h>

std::recursive_mutex g_mutex;
std::mutex g_log;

void log(std::string msg) {
  std::scoped_lock lk(g_log);
  std::cout << "[" << std::this_thread::get_id() << "] " << msg << std::endl;
}

void run_in_main_thread() {
  log("Enter run_in_main_thread.");

  {
    log("accquiring lock...");
    std::scoped_lock lk(g_mutex);
    log("lock accquired.");
  }
  log("lock released.");
}

void worker_proc() {
  log("Enter worker_proc.");

  emscripten_async_run_in_main_runtime_thread(EM_FUNC_SIG_V,
                                              &run_in_main_thread);
}

int main() {
  log("Enter main.");
  std::thread worker(&worker_proc);

  std::chrono::milliseconds sleep_time(10);
  {
    log("accquiring lock...");
    std::scoped_lock lk(g_mutex);
    log("lock accquired.");

    std::this_thread::sleep_for(sleep_time);
  }
  log("lock released.");

  worker.join();
  return 0;
}

Typical output is:

1: [16324] Enter main.
2: [16324] accquiring lock...
3: [16324] lock accquired.
4: [5266552] Enter worker_proc.
5: [16324] Enter run_in_main_thread.
6: [16324] accquiring lock...
7: [16324] lock accquired.
8: [16324] lock released.
9: [16324] lock released.

As you can see, in line 3, main thread acquired the lock and started to sleep. In line 5 the same main thread started to do something else. If the recursive_mutex in the code were replaced with mutex, the program will deadlock.

Version of emscripten/emsdk:

emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 3.1.21 (f9e81472f1ce541eddece1d27c3632e148e9031a)
Copyright (C) 2014 the Emscripten authors (see AUTHORS.txt)
This is free and open source software under the MIT license.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Full link command and output with -v appended:

$ emcc -std=c++17 -pthread -s PTHREAD_POOL_SIZE=2 -s ENVIRONMENT=web,worker main.cc -o index.html -v
 "/Users/jiulongw/w/emsdk/upstream/bin/clang" --version
 "/Users/jiulongw/w/emsdk/upstream/bin/clang++" -target wasm32-unknown-emscripten -fignore-exceptions -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -D__EMSCRIPTEN_SHARED_MEMORY__=1 -DEMSCRIPTEN -D__EMSCRIPTEN_major__=3 -D__EMSCRIPTEN_minor__=1 -D__EMSCRIPTEN_tiny__=21 -I/Users/jiulongw/w/emsdk/upstream/emscripten/cache/sysroot/include/SDL --sysroot=/Users/jiulongw/w/emsdk/upstream/emscripten/cache/sysroot -Xclang -iwithsysroot/include/compat -std=c++17 -pthread -v -matomics -mbulk-memory main.cc -c -o /var/folders/q0/bv2y31zx13x0wk6r7wvbkn100000gn/T/emscripten_temp_j4p8r721/main_0.o
clang version 16.0.0 (https://github.com/llvm/llvm-project a4a29438f451370ed241dde30bfcaab0fdf2ab71)
Target: wasm32-unknown-emscripten
Thread model: posix
InstalledDir: /Users/jiulongw/w/emsdk/upstream/bin
 (in-process)
 "/Users/jiulongw/w/emsdk/upstream/bin/clang-16" -cc1 -triple wasm32-unknown-emscripten -emit-obj -mrelax-all --mrelax-relocations -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name main.cc -mrelocation-model static -mframe-pointer=none -ffp-contract=on -fno-rounding-math -mconstructor-aliases -target-feature +atomics -target-feature +bulk-memory -target-feature +mutable-globals -target-feature +sign-ext -target-cpu generic -target-feature +atomics -target-feature +bulk-memory -mllvm -treat-scalable-fixed-error-as-warning -debugger-tuning=gdb -v -fcoverage-compilation-dir=/Users/jiulongw/w/emscripten-mutex-test -resource-dir /Users/jiulongw/w/emsdk/upstream/lib/clang/16.0.0 -D __EMSCRIPTEN_SHARED_MEMORY__=1 -D EMSCRIPTEN -D __EMSCRIPTEN_major__=3 -D __EMSCRIPTEN_minor__=1 -D __EMSCRIPTEN_tiny__=21 -I /Users/jiulongw/w/emsdk/upstream/emscripten/cache/sysroot/include/SDL -isysroot /Users/jiulongw/w/emsdk/upstream/emscripten/cache/sysroot -internal-isystem /Users/jiulongw/w/emsdk/upstream/emscripten/cache/sysroot/include/wasm32-emscripten/c++/v1 -internal-isystem /Users/jiulongw/w/emsdk/upstream/emscripten/cache/sysroot/include/c++/v1 -internal-isystem /Users/jiulongw/w/emsdk/upstream/lib/clang/16.0.0/include -internal-isystem /Users/jiulongw/w/emsdk/upstream/emscripten/cache/sysroot/include/wasm32-emscripten -internal-isystem /Users/jiulongw/w/emsdk/upstream/emscripten/cache/sysroot/include -std=c++17 -fdeprecated-macro -fdebug-compilation-dir=/Users/jiulongw/w/emscripten-mutex-test -ferror-limit 19 -fvisibility=default -pthread -fgnuc-version=4.2.1 -fcxx-exceptions -fignore-exceptions -fexceptions -fcolor-diagnostics -iwithsysroot/include/compat -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -o /var/folders/q0/bv2y31zx13x0wk6r7wvbkn100000gn/T/emscripten_temp_j4p8r721/main_0.o -x c++ main.cc
clang -cc1 version 16.0.0 based upon LLVM 16.0.0git default target x86_64-apple-darwin21.6.0
ignoring nonexistent directory "/Users/jiulongw/w/emsdk/upstream/emscripten/cache/sysroot/include/wasm32-emscripten/c++/v1"
ignoring nonexistent directory "/Users/jiulongw/w/emsdk/upstream/emscripten/cache/sysroot/include/wasm32-emscripten"
#include "..." search starts here:
#include <...> search starts here:
 /Users/jiulongw/w/emsdk/upstream/emscripten/cache/sysroot/include/SDL
 /Users/jiulongw/w/emsdk/upstream/emscripten/cache/sysroot/include/compat
 /Users/jiulongw/w/emsdk/upstream/emscripten/cache/sysroot/include/c++/v1
 /Users/jiulongw/w/emsdk/upstream/lib/clang/16.0.0/include
 /Users/jiulongw/w/emsdk/upstream/emscripten/cache/sysroot/include
End of search list.
 "/Users/jiulongw/w/emsdk/upstream/bin/wasm-ld" -o index.wasm /var/folders/q0/bv2y31zx13x0wk6r7wvbkn100000gn/T/emscripten_temp_j4p8r721/main_0.o -L/Users/jiulongw/w/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten /Users/jiulongw/w/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/crtbegin.o -lGL-mt -lal -lhtml5 -lstubs-debug -lnoexit -lc-mt-debug -ldlmalloc-mt -lcompiler_rt-mt -lc++-mt-noexcept -lc++abi-mt-noexcept -lsockets-mt -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr --import-undefined --import-memory --shared-memory --strip-debug --export-if-defined=main --export-if-defined=_emscripten_thread_init --export-if-defined=_emscripten_thread_exit --export-if-defined=_emscripten_thread_crashed --export-if-defined=_emscripten_tls_init --export-if-defined=pthread_self --export-if-defined=__start_em_asm --export-if-defined=__stop_em_asm --export-if-defined=__start_em_js --export-if-defined=__stop_em_js --export-if-defined=__main_argc_argv --export-if-defined=fflush --export=emscripten_stack_get_end --export=emscripten_stack_get_free --export=emscripten_stack_get_base --export=emscripten_stack_init --export=stackSave --export=stackRestore --export=stackAlloc --export=__wasm_call_ctors --export=__errno_location --export=emscripten_dispatch_to_thread_ --export=_emscripten_thread_free_data --export=emscripten_main_browser_thread_id --export=emscripten_main_thread_process_queued_calls --export=emscripten_run_in_main_runtime_thread_js --export=emscripten_stack_set_limits --export=__get_temp_ret --export=__set_temp_ret --export=malloc --export=free --export-table -z stack-size=5242880 --initial-memory=16777216 --no-entry --max-memory=16777216 --global-base=1024
 "/Users/jiulongw/w/emsdk/upstream/bin/wasm-emscripten-finalize" --dyncalls-i64 --pass-arg=legalize-js-interface-exported-helpers index.wasm -o index.wasm --detect-features
 "/Users/jiulongw/w/emsdk/node/14.18.2_64bit/bin/node" /Users/jiulongw/w/emsdk/upstream/emscripten/src/compiler.js /var/folders/q0/bv2y31zx13x0wk6r7wvbkn100000gn/T/tmp5767gmmo.json
 "/Users/jiulongw/w/emsdk/upstream/bin/llvm-objcopy" index.wasm index.wasm --remove-section=.debug* --remove-section=producers
 "/Users/jiulongw/w/emsdk/node/14.18.2_64bit/bin/node" /Users/jiulongw/w/emsdk/upstream/emscripten/tools/preprocessor.js /var/folders/q0/bv2y31zx13x0wk6r7wvbkn100000gn/T/emscripten_temp_j4p8r721/settings.js worker.js --expandMacros
 "/Users/jiulongw/w/emsdk/node/14.18.2_64bit/bin/node" /Users/jiulongw/w/emsdk/upstream/emscripten/tools/preprocessor.js /var/folders/q0/bv2y31zx13x0wk6r7wvbkn100000gn/T/emscripten_temp_j4p8r721/settings.js shell.html

[sbc100]

Whats happening here is not so much that the main thread is being re-entered, but that the main thread runs its work queue while its sleeping. The main thread never really blocks, it just busy loops and runs any incoming work.

The work queue is run whenever the main thread yields. See emscripten_yield: https://github.com/emscripten-core/emscripten/blob/fdb1656ee89b21b9e238ce1b7f76b324c795aaac/system/lib/pthread/emscripten_yield.c#L42 (which happens during sleep or when waiting on a mutex).

[jiulongw]

Thanks for the clarification @sbc100 . This is by design then. But it may cause many unexpected behaviors. For example, one can never expect that main thread work queue is executed in order, because a work item can start before the previous one finishes.

[tlively]

You can control the proxying to the main thread more directly by using the proxying.h API rather than the higher-level proxying APIs.

[sbc100]

Thanks for the clarification @sbc100 . This is by design then. But it may cause many unexpected behaviors. For example, one can never expect that main thread work queue is executed in order, because a work item can start before the previous one finishes.

I believe the main work queue is executed in order and is not re-entrant: https://github.com/emscripten-core/emscripten/blob/0ca0db492609557794c9dff7b4930b017c952ba2/system/lib/pthread/proxying.c#L363-L368

[jamesruan]

in our use case, we have the following stack trace of a dead-locked situation: It seems that the main thread yield when doing a dlmalloc ?

The callback in line 21 is reenter when line 34 is mallocing?

As a normal developer, I understand that the main thread runs the task queue when sleeping but not expecting it happens in a simple map insertion.