go-tableland icon indicating copy to clipboard operation
go-tableland copied to clipboard

[GOT-31] Explore if we can make an iterator-based read-query execution pipeline

Open jsign opened this issue 2 years ago • 0 comments

This is capturing an idea that I had a while ago. Currently, we store all the read-query results in memory and do the right formating after.

That might be comfortable for formatting but also means that we hold all the read-query results in memory which is very aggressive to the validator. Users could send queries that have very big results, and that will directly impact memory usage.

We should explore if we can leverage the iteration we do with rows.Next() in the lowest layer and pass that upstream to do the right formatting to have a bounded memory usage.

cc @brunocalza @sanderpick @asutula

GOT-31

jsign avatar Sep 30 '22 18:09 jsign