help icon indicating copy to clipboard operation
help copied to clipboard

worker_threads is used by many people,but we can only use SharedArrayBuffer(for shared memory) ,but Buffer is node's first citizen(socket eg),can you share Buffer to SharedArrayBuffer ?

Open introspection3 opened this issue 2 years ago • 4 comments

What is the problem this feature will solve?

threads_worker is used by many people,but we can only use SharedArrayBuffer ,but Buffer is node's first citizen(socket eg),can you share Buffer to SharedArrayBuffer ?

What is the feature you are proposing to solve the problem?

let instance=SharedArrayBuffer.from(Buffer instance);//zero copy let instance2=Buffer.From(SharedArrayBuffer instance);//zero copy

What alternatives have you considered?

No response

introspection3 avatar Nov 07 '23 01:11 introspection3

You mean node:worker_threads, right? If all buffers could be shared, the language would not have bothered to come up with a SharedArrayBuffer implementation. What you can do is the following:

function sabToBuffer(sab) {
  return new Uint8Array(sab);
}

Uint8Array is not exactly a Buffer, but I don't think you'll find any Node.js API that accept Buffer and not Uint8Array (if you do, that'd be a bug, please report it).

On the other hand, if you have a Buffer/Uint8Array and want to transfer its content to an other thread, you have two options:

// Option 1: copy the content to a SharedArrayBuffer:
function copyBufferToSAB(buffer, sab) {
  if (buffer.byteLength > sab.byteLength) throw new Error('Buffer overflow');
  new Uint8Array(sab).set(buffer);
}

// Option 2: transfer the underlying array buffer:
function transferBuffer(buffer) {
  otherThread.postMessage(buffer, [buffer.buffer]);
}

Option 2 is way faster, but note that only one thread interact with a non-sharable buffer, meaning that once it is transfered, the original thread can no longer have access to that buffer data.

aduh95 avatar Nov 07 '23 08:11 aduh95

@aduh95 no,you are wrong. copy ArrayBuffer is so expensive。

introspection3 avatar Nov 07 '23 09:11 introspection3

@aduh95 do you know zero copy

introspection3 avatar Nov 07 '23 09:11 introspection3

no,you are wrong. copy ArrayBuffer is so expensive

I don't know why you say I'm wrong, I don't think I have said anything about copy not being expensive. Like I said, transfering is the faster option, that's the one that let you reuse the same data on another thread without copying data.

aduh95 avatar Nov 07 '23 10:11 aduh95

@introspection3 @aduh95 is this issue resolved? If so, feel free to close the issue!

Thanks!

avivkeller avatar Jun 15 '24 12:06 avivkeller