pg_duckdb icon indicating copy to clipboard operation
pg_duckdb copied to clipboard

Parallel Scan with multiple processes

Open mkaruza opened this issue 1 year ago • 2 comments

  • We can now start multiple worker process that can read relation blocks and write buffer to shared memory to be consumed by duckdb reader threads. Problem can be viewed as variant of multi producer / multi consumer where producer are responsible for releasing buffer after they are read.

mkaruza avatar Jul 02 '24 07:07 mkaruza

@mkaruza what's the deal with this? I guess it's not critical for 0.1.0. Given the amount of merge conflicts it has now, I'm think it probably makes the most sense to simply close this and maybe create an issue for it instead.

JelteF avatar Sep 30 '24 12:09 JelteF

@JelteF agree not critical, but could be approach to get better performances of heap table scans (specifically to fetch table pages in parallel; currently there is bottleneck there as only one thread in process can request page fetching).

mkaruza avatar Sep 30 '24 14:09 mkaruza

@mkaruza do you want to close this one in favor of #477

JelteF avatar Dec 09 '24 09:12 JelteF

Closing

mkaruza avatar Dec 09 '24 15:12 mkaruza