node-rdkafka icon indicating copy to clipboard operation
node-rdkafka copied to clipboard

Fix worker_threads support

Open Qard opened this issue 11 months ago • 2 comments

This makes node-rdkafka work correctly with worker_threads. There were several issues which needed resolving and some changes which were needed to prevent data being shared across threads and to ensure everything is cleaned up completely when the thread exits.

Changes included:

  • Use NODE_MODULE_CONTEXT_AWARE to init module
  • No globals, shared data needs to live in PerIsolateData.
  • No using uv_default_loop(), need to get the loop owned by the current thread.
  • Dispatcher::Deactivate() on destruct to stop uv_async_t so thread can teardown loop safely
  • Connection and Topic types need cleanup hooks to ensure they are deleted when the thread is torn down.
  • Construct Conf with correctly-sized type so destructor and sub-class fields aren't overwriting random memory
  • Give AsyncWorker types unique AsyncResource names for better debugging

Qard avatar Dec 20 '24 14:12 Qard

@GaryWilber can you take a look?

mcollina avatar Jan 07 '25 10:01 mcollina

Currently, no native-backed Node.js Kafka library seems to support worker threads...

DrJume avatar Apr 19 '25 19:04 DrJume