bayeslite icon indicating copy to clipboard operation
bayeslite copied to clipboard

Slowdown in Bayeslite or Crosscat

Open tibbetts opened this issue 9 years ago • 2 comments

I’m seeing 0.1.4 being notably slower on my laptop than 0.1.3.rc2. Specifically running the following queries:

%time q('''simulate Period_Minutes from satellites_cc GIVEN "Users" = 'Military' limit 1000''');

takes 539ms on 0.1.3rc2 and 21s on 0.1.4 in unscientific testing. The underlying crosscat versions are 0.1.38 and 0.1.49 respectively.

tibbetts avatar Jan 12 '16 00:01 tibbetts

Profile run with:

import cProfile
cProfile.run("""q('simulate Period_Minutes from satellites_cc GIVEN "Users" = Military limit 1000')""")

bayeslite-0.1.4-profile.txt bayeslite-0.1.3rc2-profile.txt

tibbetts avatar Jan 12 '16 00:01 tibbetts

All the time seems to be spend inside cursor.execute(), in sqlite3/apsw.

A few wild hypotheses to investigate:

  • apsw's statement cache is problematic -- to large or too small. We could try passing statementcachesize=0 or statementcachesize=1000 to apsw.Connection(...).
  • apsw is being compiled with -O0 instead of -O2 for some reason.
  • apsw is acquiring locks or doing other interprocessor synchronization that the Python sqlite3 module doesn't.

riastradh-probcomp avatar Jan 12 '16 00:01 riastradh-probcomp