PySyft icon indicating copy to clipboard operation
PySyft copied to clipboard

[wip] remove connection pool from sqlite

Open abyesilyurt opened this issue 1 year ago • 1 comments

Sqlite connection is not thread-safe. We use complicated and unstable pooling to make connections thread-safe. We don't need to do any locking if we just open the connection per each thread.

The tests are also much more stable now.

abyesilyurt avatar May 09 '24 12:05 abyesilyurt

Could def _cursor just yield con.cursor() instead of

cur = con.cursor()
yield cur.execute(sql, *args)

?

We can rewrite this

with self._cursor(
    insert_sql, [str(key), _repr_debug_(value), data, str(key)]
) as _:
    pass

into

with self._cursor as cursor:
    cursor.execute(insert_sql, [str(key), _repr_debug_(value), data, str(key)])

looks a bit more idiomatic to me.

I agree that it looks better, but I think this change is out of scope for this PR as it leads to much bigger refactoring. I will create an issue for this one though.

abyesilyurt avatar Jun 03 '24 08:06 abyesilyurt

I figured out some problems with this implementation and database locked errors are still there, when several jobs are launched.

abyesilyurt avatar Jul 01 '24 10:07 abyesilyurt