node-postgres icon indicating copy to clipboard operation
node-postgres copied to clipboard

Readable never recieving 'end' event on COPY stream

Open Ivor808 opened this issue 1 year ago • 0 comments

Hey all,

I'm using node pg + the copy extension to run COPY TO STDOUT on the postgres server to write to a file locally. I've noticed that certain tables and row combinations result in the readable stream never recieving the "end" event (even when data events stop coming through). Thus the stream hangs open forever. I then checked the pg_stat_progress_copy table and that had 0 results even when the stream was still hanging. I then checked the pg_stat_activity and can see the copy process in there in an idle state. The data size in my repo case is 300 lines from the DB aka very small. I actually noticed this issue with the postgresjs package too. So I'm not sure if its specific to nodepg or postgres.

Here is an example of my COPY code

 const client = await pgPool.connect();
  const readable = client.query(copyTo(copyQuery));

  try {
    await pipeline(
      readable,
      createWriteStream(`${os.homedir()}/cms/${fileName}`)
    );
  } finally {
    client.release();
  }

Here I attached end and close event listeners to the readable stream. You can see the first "downloading" gets a end and close event. While the second readable stream does not and hangs. image

Edit: I'm noticing it actually getting stuck on the same table which never sends the close event but doesnt read any more data in

Thank you.

Ivor808 avatar Jul 26 '24 00:07 Ivor808