Worker in a worker and other parallelization architecture examples

[ascandal]

I am running service what launches its own master node worker and that master worker needs to launch other slave worker nodes.

I'm interested in seeing this use case example and other multi worker setups.

In this particular implementation, I'm trying to create a pool of slave workers from within the master worker.

Running into webpack issues at the moment.

Critical dependency: require function is used in a way in which dependencies cannot be statically extracted

Critical dependency: the request of a dependency is an expression

require.extensions is not supported by webpack. Use a loader instead.

Any thoughts or comments?

I can post a minimal example to illustrate my initial appoach.

[ascandal]

Does this work?

Example - Master Worker launches Slave Pool

The main server that launches the master worker.

// Server.js
import { spawn, Thread, Worker } from "threads"

async function main() {
  const masterProcess = await spawn(new Worker("./workers/master.worker"))
  const allSlaveResults = await masterProcess(params)

  console.log(`master process complete`)

  await Thread.terminate(add)
}

main().catch(console.error)

The master worker program that launches the pool of slave workers.

// workers/master.worker.js
import { expose } from "threads/worker"
import { spawn, Pool, Worker } from "threads"

expose(function masterProcess(params) {
    const pool = Pool(() => spawn(new Worker("./workers/slave.worker")), 8 /* optional size */)

    const allSlaveResults = [];

    // processing to create slave params.
    const slaveParams = createSlaveParams(params);

    let indexSlaveParams = 0;

    pool.queue(async slaveProcess => {
        const slaveResult = await slaveProcess(slaveParams[indexSlaveParams++]);
        allSlaveResults.push(slaveResult);
        console.log(`slave process complete`)
    })

    await pool.completed()
    await pool.terminate()

    return allSlaveResults;
})

The slave worker program that does distributed work.

// workers/slave.worker.js
import { expose } from "threads/worker"

expose(function slaveProcess(slaveParams) {
    // do some distributed work.
    return slaveResult;
})

[andywer]

Hey @ascandal!

Sorry for the late response, somehow missed the notification.

Good question! I actually haven't tried yet. Does it work?

[ascandal]

@andywer I could not get it to work. I think its a problem with my webpack setup. Seems that when transpiling from typescript to javascript it does not like the call to new Worker() in the master worker.

[simplecommerce]

I tried to make something similar to this, I am using Parcel 2 for the building and I seem to get an error when it tries to load the slave worker.

Uncaught DOMException: Failed to execute 'importScripts' on 'WorkerGlobalScope': The URL './dataTable.cb6c71e3.js' is invalid.

I have no idea if its an issue with the bundling via Parcel or if its related to this package.

Update: Ok it must be something in my code, because I did a simple master and add function and it works so.

Update 2: Ok a weird thing is happening, I can call a slave worker from a master worker, but only 1. If I try to call 2, it gives me the above error. Here is a glimpse of the master code.

I can comment the top or botton and they run fine alone, but if I use both it fails.

import 'regenerator-runtime/runtime';
import { spawn, Thread, Worker, expose, Transfer } from "threads";

expose(async ({ columns, rows, language, translations }) => {
  const dataTable = await spawn(new Worker("./dataTable/index.js"));
  const renderedRows = await dataTable({ columns, rows, language, translations });      
  await Thread.terminate(dataTable);  
  
  const add = await spawn(new Worker("./add.js"));
  const result = await add();
  await Thread.terminate(add);
  
  return result;
});