langchain icon indicating copy to clipboard operation
langchain copied to clipboard

Published 20 hours ago • verified publisher icon langchain-ai

feat: Improvements to SQLDatabaseChain for better local model support (query checker and few shot examples)

Open tjaffri opened this issue 1 year ago • 5 comments

I have been playing around with SQLDatabaseChain using some non-OpenAI models, and found some useful tips and tricks:

  1. Use a query checker to fix small mistakes in the model generated SQL. This is the same trick used by the SQL agent, and seems to help even with the larger models.
  2. Use few shot examples to improve model output, similar to what @jzluo tweeted about here: https://twitter.com/JonZLuo/status/1638638298666004483?s=20. For this, I added more intermediate steps and also returned intermediate steps in case of exceptions (so that the caller can log and add to few shot prompt examples)
  3. If using the sequential chain, discard invalid table names which seem to be produced more often with smaller locally hosted models.

I also updated the example notebook with some ideas on how to collect few shot examples in production.

I ran the following locally:

  1. make format
  2. make lint
  3. make test

Please let me know if there is any other formality I need to complete... first PR to this awesome project :)

image

tjaffri avatar Apr 27 '23 05:04 tjaffri

@hwchase17 @vowelparrot @agola11 @PeadarOhAodha @nfcampos I tried to rebase this to match your PR #3256. Would appreciate a review of this PR as well, and any suggestions for improvement.

BTW @jzluo this includes some of your ideas from the tweet in the description so would appreciate a quick look from you as well.

Thank you folks!

tjaffri avatar May 01 '23 04:05 tjaffri

@hwchase17 @vowelparrot @agola11 @PeadarOhAodha @nfcampos review please?

tjaffri avatar May 02 '23 19:05 tjaffri

@hwchase17 @vowelparrot @agola11 @PeadarOhAodha @nfcampos review please?

sorry for delay - will try to take a look in next day or so!

hwchase17 avatar May 03 '23 22:05 hwchase17

@hwchase17 you want me to use the new PR template that just went in? Let me know what I can do to make your review easier... I know you folks are super busy and I'm in awe of how much code you have been shipping :)

Thank you!

tjaffri avatar May 09 '23 00:05 tjaffri

@hwchase17 I've been trying to use non-OpenAI models with SQLDatabaseChain and these changes helped me overcome the problems I'd been running into. It would be awesome to see this merged to help others with this issue!

Thanks for all you've done with this project!!

dglindner2 avatar May 10 '23 20:05 dglindner2