sqlalchemy object caching issue

[dotX12]

Greetings! I ran into sqlalchemy object caching issues. From time to time I get errors like this:

"  File \"/home/boot/stellar/backend/./app/v1/security/auth.py\", line 127, in get_current_user\n    permissions_v2 = await service_permissions.get_permissions_by_role_cache(\n",
      "  File \"/home/boot/stellar/backend/.venv/lib/python3.10/site-packages/cashews/wrapper.py\", line 272, in _call\n    return await decorator(*args, **kwargs)\n",
      "  File \"/home/boot/stellar/backend/.venv/lib/python3.10/site-packages/cashews/decorators/cache/simple.py\", line 43, in _wrap\n    await backend.set(_cache_key, result, expire=ttl)\n",
      "  File \"/home/boot/stellar/backend/.venv/lib/python3.10/site-packages/cashews/validation.py\", line 62, in _invalidate_middleware\n    return await call(*args, key=key, **kwargs)\n",
      "  File \"/home/boot/stellar/backend/.venv/lib/python3.10/site-packages/cashews/wrapper.py\", line 34, in _auto_init\n    return await call(*args, **kwargs)\n",
      "  File \"/home/boot/stellar/backend/.venv/lib/python3.10/site-packages/cashews/disable_control.py\", line 12, in _is_disable_middleware\n    return await call(*args, **kwargs)\n",
      "  File \"/home/boot/stellar/backend/.venv/lib/python3.10/site-packages/cashews/backends/client_side.py\", line 132, in set\n    return await super().set(self._prefix + key, value, *args, **kwargs)\n",
      "  File \"/home/boot/stellar/backend/.venv/lib/python3.10/site-packages/cashews/serialize.py\", line 107, in set\n    value = pickle.dumps(value, protocol=pickle.HIGHEST_PROTOCOL, fix_imports=False)\n",
      "_pickle.PicklingError: Can't pickle <function __init__ at 0x7fc386fbb7f0>: it's not the same object as sqlalchemy.orm.instrumentation.__init__\n"

I found the answer that such objects should be serialized and deserialized differently: https://docs.sqlalchemy.org/en/14/core/serializer.html

Tell me how good / bad idea it is to cache alchemy objects, and not the endpoint itself.

If this is not bad, what can you say about adding functionality that could fix this error?

[Krukov]

Hello,

I think objects like sqlalchemy orm is a good use case for caching, if it possible to pickle somehow. So Yep, cashews should support it ( thanks for the link), I am going to fix this issue ASAP.

[Krukov]

Hi @dotX12 , the release is ready to upload on pypi but I can't reproduce this issue. ) I tried with simple examples but with different types of sqlalchemy objects and can't get an error like you have.

import asyncio

from cashews import cache
cache.setup("redis://")

from sqlalchemy import Column, Integer, String
from sqlalchemy.orm import declarative_base, sessionmaker
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.future import select
from sqlalchemy.ext.asyncio import create_async_engine

Base = declarative_base()


class User(Base):
    __tablename__ = 'users'
    id = Column(Integer, primary_key=True)
    name = Column(String)
    fullname = Column(String)
    nickname = Column(String)


@cache(ttl="1m", key="user:{id}")
async def get_user(session, id):
    user = await session.get(User, id)
    return user


async def main():
    engine = create_async_engine('sqlite+aiosqlite:///:memory:', echo=False)
    async with engine.begin() as conn:
        await conn.run_sync(Base.metadata.create_all)

    async_session = sessionmaker(
        engine, expire_on_commit=False, class_=AsyncSession
    )
    async with async_session() as session:
        async with session.begin():
            session.add_all(
                [
                    User(name="test", id=11, fullname="10 th", nickname="test10"),
                ]
            )
        user = await get_user(session, 11)
        print(user.fullname)


asyncio.run(main())
    

[dotX12]

Привет, @dotX12, релиз готов к загрузке на pypi, но я не могу воспроизвести эту проблему. ) Я попробовал с простыми примерами, но с разными типами объектов sqlalchemy и не могу получить ошибку, как у вас.

import asyncio

from cashews import cache
cache.setup("redis://")

from sqlalchemy import Column, Integer, String
from sqlalchemy.orm import declarative_base, sessionmaker
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.future import select
from sqlalchemy.ext.asyncio import create_async_engine

Base = declarative_base()


class User(Base):
    __tablename__ = 'users'
    id = Column(Integer, primary_key=True)
    name = Column(String)
    fullname = Column(String)
    nickname = Column(String)


@cache(ttl="1m", key="user:{id}")
async def get_user(session, id):
    user = await session.get(User, id)
    return user


async def main():
    engine = create_async_engine('sqlite+aiosqlite:///:memory:', echo=False)
    async with engine.begin() as conn:
        await conn.run_sync(Base.metadata.create_all)

    async_session = sessionmaker(
        engine, expire_on_commit=False, class_=AsyncSession
    )
    async with async_session() as session:
        async with session.begin():
            session.add_all(
                [
                    User(name="test", id=11, fullname="10 th", nickname="test10"),
                ]
            )
        user = await get_user(session, 11)
        print(user.fullname)


asyncio.run(main())
    

Hi, I'm not at home right now, I'll test it tomorrow evening, thanks!

[Krukov]

OK but you can test it only on master branch - release postponed. It will be nice to have and example of code with this bug

[dotX12]

@Krukov, indeed, your example doesn't cause this problem. I can't share my example that causes this problem (the problem persists and hasn't been fixed :c), as it is the project with NDA. but I'll try to make a test case that reproduces it.

[Krukov]

@dotX12 Hello Do you have some examples?

[dotX12]

@Krukov Hello, I have now gone to the sea, I think I should not have opened this issue so early :) I think I can only answer the august 20 to you :) Sry

[Krukov]

Don't worry, it is OK. Have a good weekend!

[Krukov]

Hi @dotX12 . Do you have any updates? )

[dotX12]

Hi @dotX12 . Do you have any updates? )

Hello, now I sit down to reproduce the bug, I think tomorrow or after tomorrow I will send it!

[dotX12]

@Krukov, pickletype="sqlalchemy" really solved my problem, I don't know why it didn't work last time, maybe I didn't update the library version to master, but now the problem is gone!

solved :)

[Krukov]

Great, than I will release new version soon

[Krukov]

released with 4.7.1