zserio icon indicating copy to clipboard operation
zserio copied to clipboard

SQLite API: Make Row Reader more efficient and backward compatible

Open fklebert opened this issue 1 year ago • 1 comments

At the moment the zserio generated C++ API for Rows of a SQLite table does kind of a "SELECT *" from the relevant table. It does not really use "*", but all the column names that are available in the schema. This is in two cases not optimal:

  1. It makes SQLite fetch all columns, which in many cases is not necessary. In case of large blobs in columns, it will generate unnessecary memory pressure and performance degradation.
  2. It breaks backward compatibility with older databases in case columns have been added in subsequent versions.

It would be beneficial if we can extend ::createReader(condition) to also allow to specify the colunms to be fetched. Like ::createReader(columns, condition)

This has been observed in the C++ API. I haven't checked the others...

fklebert avatar Jan 31 '24 09:01 fklebert

Yes, I agree. This is a good enhancement of SQLite API. The same problem is in Java and Python API(s).

mikir avatar Jan 31 '24 10:01 mikir