bayeslite
bayeslite copied to clipboard
Slowdown in Bayeslite or Crosscat
I’m seeing 0.1.4 being notably slower on my laptop than 0.1.3.rc2. Specifically running the following queries:
%time q('''simulate Period_Minutes from satellites_cc GIVEN "Users" = 'Military' limit 1000''');
takes 539ms on 0.1.3rc2 and 21s on 0.1.4 in unscientific testing. The underlying crosscat versions are 0.1.38 and 0.1.49 respectively.
Profile run with:
import cProfile
cProfile.run("""q('simulate Period_Minutes from satellites_cc GIVEN "Users" = Military limit 1000')""")
All the time seems to be spend inside cursor.execute(), in sqlite3/apsw.
A few wild hypotheses to investigate:
- apsw's statement cache is problematic -- to large or too small. We could try passing
statementcachesize=0
orstatementcachesize=1000
toapsw.Connection(...)
. - apsw is being compiled with
-O0
instead of-O2
for some reason. - apsw is acquiring locks or doing other interprocessor synchronization that the Python sqlite3 module doesn't.