aiopg icon indicating copy to clipboard operation
aiopg copied to clipboard

Add cache for precompiled SA queries

Open asvetlov opened this issue 11 years ago • 4 comments

asvetlov avatar Apr 24 '14 17:04 asvetlov

How do you want to organise it without name collision? . I use model dict (name -> select) for custom selectors: I use bindparam() for arguments and store these dicts locally..

eirnym avatar Apr 29 '15 13:04 eirnym

I think LRU cache for expressions may help. ClauseElement has method .compare(other) BTW.

So query like conn.execute(tbl.insert(), {'a': 1, 'b': 2}) will be cached fine, in the same cell as conn.execute(tbl.insert(), {'a': 3, 'b': 4}).

But conn.execute(tbl.insert().values(a=1, b=2)) is bad candidate to cache, LRU will kick it off quickly.

But I still have no evidence that SQLAlchemy query compiling is a bottleneck.

asvetlov avatar May 06 '15 19:05 asvetlov

I've done benchmark to allow execute precompiled queries. I've got:

aiopg (sql): 182446 queries in 30 seconds aiopg_sa: 88508 queries in 30 seconds aiopg_sa_cache: 118051 queries in 30 seconds

So if this issue will be done, it can give about 25% to speedup.

VelikiiNehochuha avatar Nov 27 '17 12:11 VelikiiNehochuha

Unlimited cache size bothers me. Maybe better to use lru_cache?

@functools.lru_cache(256)
def get_compiled(query):
    return query.compile()

asvetlov avatar Dec 06 '17 10:12 asvetlov