deltachat-core-rust icon indicating copy to clipboard operation
deltachat-core-rust copied to clipboard

sql: enable auto_vacuum on all connections

Open link2xt opened this issue 3 years ago • 7 comments

Trying to enable auto_vacuum on all connections

link2xt avatar Jan 06 '22 21:01 link2xt

This is flaky, I originally pushed it to master as ce0984f02f62b72afe5db7745f152447740e760a after testing, but reverted because of "database is busy" errors on CI: 652d67a20f8eaa8dcaa6fa93618ad6d2a48ba112

This is probably working correctly, but is slower because of this.

link2xt avatar Jan 06 '22 21:01 link2xt

The point of having this pragma enabled on all connections is to ensure that when we execute VACUUM, no matter on which connection it happens, we enable auto_vacuum.

This can't be merged as is though, it fails random tests with "database is locked" error on CI currently.

link2xt avatar Jan 06 '22 22:01 link2xt

This can't be merged as is though, it fails random tests with "database is locked" error on CI currently.

wondering what is happening here. databases used for ci are only a few kb large and even a full VACUUM should take only a ms or so.

r10s avatar Jan 06 '22 23:01 r10s

Auto vacuum is not really important right now, it is enabled for new databases (with a test for it) and for existing databases if it's not enabled worst case is that the database never shrinks in size except on VACUUM, but the space is still reused by the database itself.

link2xt avatar Jan 07 '22 00:01 link2xt

Dropping sqlite connection pool does not result in actually closing connections immediately. On windows this is a problem, because files remain locked for some time after drop. As a workaround, migration test and account removal now has a 5 second sleep to ensure all connections are actually closed :/

There is an open bugreport about this in r2d2: https://github.com/sfackler/r2d2/issues/99

link2xt avatar Apr 16 '22 22:04 link2xt

I guess as a workaround when we try to rename files or delete files, we should sleep 1 second and retry up to 5 times or something like this.

link2xt avatar Apr 16 '22 22:04 link2xt

Waiting for #3229 to be merged and a similar fix for migrate_account is needed.

link2xt avatar Apr 17 '22 23:04 link2xt

Auto vacuum is not really important right now, it is enabled for new databases (with a test for it) and for existing databases if it's not enabled worst case is that the database never shrinks in size except on VACUUM, but the space is still reused by the database itself.

If this is still true and the PR triggers issues in tests the i think this can be put to project resurrection. It would become more important if we put blob files into the same database used for messages, networking etc. And then only for older databases. New databases are created with auto-vaccum as noted in the quote.

hpk42 avatar Dec 04 '22 19:12 hpk42

Waiting for #3229 to be merged and a similar fix for migrate_account is needed.

I implemented retries for account migration in #4043.

link2xt avatar Feb 15 '23 15:02 link2xt

Now with #4043 tests pass.

link2xt avatar Feb 15 '23 16:02 link2xt