postgres icon indicating copy to clipboard operation
postgres copied to clipboard

[feature] Support streaming of buffer/bytea parameters

Open vicary opened this issue 3 years ago • 0 comments

My app manages deep learning models and it's metadata, the models are sized around 50MB - 100MB.

I am using CockroachDB for storage because it has no hard limit on their BYTES / BYTEA / BLOB columns, it is handy when the binary data is infrequently accessed. The binary data is for records and statistical study, they are usually written once and rarely read.

I know it is not the best practice out there, but I don't want to setup object storage just for them.

With the current version, when the buffer goes more than ~30MB V8 will start throwing heap allocation failures, I believe https://github.com/denoland/rusty_v8/pull/427 is not enough to catch up with the sudden memory spike.

Does it make sense if the parameters can be transferred via ReadableStream and WriteableStream?

vicary avatar Sep 18 '22 18:09 vicary