chainlit icon indicating copy to clipboard operation
chainlit copied to clipboard
verified publisher icon Chainlit

[WIP] - SQL Model Data Layer

Open hayescode opened this issue 7 months ago • 5 comments

[!CAUTION] This is currently a mockup and incomplete. This has not been tested and will not run.

This is an initial draft for consolidating all SQL data layers using FastAPI's sqlmodel (Pydantic + SQL Alchemy).

For now I have created the new models in a new models/ directory and are duplicated from step.py, element.py, etc. The idea would be to delete those once this is ready.

The main reasons for this change are:

  1. Enable everyone who wants to use SQL data persistence to use a single implementation.
  2. Reduce technical debt and bugs, and speed up future development.

replaces #1365

Notes:

  • sqlmodel reserves the 'metadata' field for internal purposes and cannot be changed, so in Threads/Steps/User we have to rename this to something else so for now it's metadata_ instead, then aliased for proper database and API usage.

hayescode avatar Aug 29 '25 19:08 hayescode

This PR is stale because it has been open for 14 days with no activity.

github-actions[bot] avatar Sep 18 '25 02:09 github-actions[bot]

@github-actions Leave this PR open

asvishnyakov avatar Sep 18 '25 12:09 asvishnyakov

How can I support in this pull request? I was on holiday and then almost forgot this PR @hayescode

niklasdiehm avatar Oct 09 '25 07:10 niklasdiehm

@niklasdiehm here's the thread on it in discord (can also be found in gh issues)

https://discord.com/channels/1088038867602526210/1421299306404647012

I've been very busy at work lately and will probably continue to be. Sorry I couldn't make more progress here.

I think we need to align on a design and structure then we can split into pieces with this PR being the target branch. I think sqlmodel would be great (FastAPI + SQLAlchemy + Pydantic) but I'm not married to it. Things like the "metadata" column are a reserved word and requires work arounds. What are your thoughts?

We'll take all of the help we can get so whatever piece of this you'd like we welcome contributions on. Thanks!

hayescode avatar Oct 09 '25 10:10 hayescode

We’ve also some odd behavior with the official data layer regarding metadata. It appears that metadata for steps is empty. attributes such as language or defaultOpen aren’t saved as attributes, and I expected them to be stored in metadata instead. However, they’re not present there either, meaning they aren’t persisted at all.

jsprpl avatar Oct 13 '25 13:10 jsprpl