postgres icon indicating copy to clipboard operation
postgres copied to clipboard

Copy stream hangs if using gzip stream compression

Open uasan opened this issue 3 years ago • 4 comments

Hello, we caught a cool bug )

Streams work fine when copying, but if the stream is sent to the compression pipe, the entire pool of connections freezes.

Example, playback:

import postgres from 'postgres';
import { createGzip } from 'zlib';

const db = postgres({
  max: 2,
});

for (let i = 0; i < 10; i++) {
  let stream = await db
    .unsafe(`COPY pg_catalog.pg_attribute TO STDOUT`)
    .readable();

  stream.pipe(createGzip()).on('data', console.log);

  await db.unsafe(`SELECT $1 AS "CONTINUE"`, [i]);
}

console.log(await db.unsafe(`SELECT 'DONE' AS "END"`));

I play with pipe, but in any case, something is wrong with the connection pool after sending the stream to the pipe.

uasan avatar Oct 07 '22 18:10 uasan

It seems that this problem was the cause of the connection errors that I described here https://github.com/porsager/postgres/issues/457. Due to the pipe, connections in the pool become busy in read mode, although postgres has returned the response and there is nothing to read

uasan avatar Oct 13 '22 13:10 uasan

Sorry I haven't been on this @uasan .. I can repro here, so I hope I'll find some time to look into it soon!

porsager avatar Jun 26 '23 00:06 porsager

I think this is due to the fact that the archiving stream sometimes pauses the stream

uasan avatar Jul 04 '23 18:07 uasan

I wonder if I am hitting a similar issue. Certain copy commands are read to stdout into a readable stream. Those readable streams sometimes just hang and it seems as if the "end" event is never transmitted.

Ivor808 avatar Jul 23 '24 04:07 Ivor808