chainlit icon indicating copy to clipboard operation
chainlit copied to clipboard

[Draft] Proposal for SQLAlchemy Data Layer with ORM

Open qtangs opened this issue 5 months ago • 5 comments

This is a proposal for a different way to implement SQLAlchemy Data Layer using ORM instead of raw SQL statements, taking inspiration from https://github.com/Chainlit/chainlit/pull/832 Hopefully it will simplify support for other SQL dialects besides Postgres and SQLite.

This is a rather major change:

  • [x] Define tables and indices using SQLAlchemy objects
  • [x] Use SQLAlchemy functions to create statements instead of raw SQL queries

Other changes:

  • [x] in get_thread and list_threads. only get specific thread(s) for better performance and reduced bandwidth compared to get_all_user_threads (note: reduced bandwidth is important for providers such as Supabase)
  • [x] in list_threads, filter would use thread name as well as steps input/output texts, this needs careful review.
  • [x] in create_element, storage_provider is made optional so that elements that don't need content to be stored will still be added to database, example of these elements are external links that can be read and displayed at runtime.

The diff between current SQLAlchemy and this one can be found here: https://gist.github.com/qtangs/71370e6ce5feec78d9586c808804e81c/revisions?diff=split&w=

Some tests have been added based on https://github.com/Chainlit/chainlit/blob/main/backend/tests/data/test_sql_alchemy.py but more might be needed.

qtangs avatar Sep 22 '24 04:09 qtangs