Memory leak

[wolfgang371]

trafficstars

require "sqlite3"

DBFILE = "test.db"

db = DB.open("sqlite3://./#{DBFILE}")
while true
    begin
        db.scalar("select padding from foo where id=#{Random.rand(100000000)}") # size matters!
    rescue DB::NoResultsError
    end
end

grows beyond 32GB of RAM on my machine in a couple of minutes.

For the setup you need: https://www.sqlite.org/src/file/ext/misc/series.c https://www.sqlite.org/loadext.html: gcc -g -fPIC -shared series.c -o series.so

save the lines below in setup.sql, then cat setup.sql | sqlite3 test.db

.load ./series
.header on
.timer on
drop table if exists foo;
create table foo (id int, padding text);
insert into foo (id, padding) select value, random() from generate_series(1, 100000000) order by random();
create index foo_id_ix on foo (id);

I use crystal 1.1.1 on Ubuntu 18.04

[wolfgang371]

the only thing that seems to help is to close and reopen the database from time to time. In this case memory isn't freed, but it seems to get reused since after the first iteration consumption stays on a constant level.

GC.collect doesn't help.

[bcardiff]

sqlite3 uses prepared statements. Always. There is a hash of statement query to statement object in crystal-db so statements are not recreated all the time.

To avoid allocating too many statements, in your example you could use:

db.scalar("select padding from foo where id=?", Random.rand(1000))

This will create only 1 statement object (per connection) since the arguments are not part of the statement query. Unless there is another source of the leak I think do it.

[wolfgang371]

Thanks a lot, works for me (no more leak)! Maybe there should be some sort of aging in the hash (or in the first step some info outputs on the console in case of excess size).