Sql database chain with retries

[bpleahey]

Adding retry functionality for SQLDatabaseChain when compile/runtime errors are encountered. For now, this only includes InvalidRequestErrors. @andersenchen @pablongo24

[andersenchen]

@bpleahey

At a high level, let's go with this structure:

for _ in range(self.max_tries):
  try:
    # generate SQL code
    # query database
    # if successful, break / return out of this loop
  except ({list your exceptions here, comma-delimited}) as e:
    # continue the for loop

Let me know if this makes sense!

[alonroth]

@bpleahey I wonder what's the status of this PR?

Its seems to me that this functionality is quite important and I wanted to see if someone is working on it before implementing it myself.

I also wonder if you're making use of the exception in the next retry so it will be addressed by the llm?

[bpleahey]

Hey Alon,I’d fallen behind on this PR because of some other projects, but I’m in a gap between what I was working on so it was honestly great timing for you to reach out. I’ll try to sync my branch later so it’s ready to keep being worked on. I can forward you some bullet points of what I’d planned to address (making use of the LLM in each retry was one of them).I’d be happy to give you edit access, as it sounds like you may be ahead of me in software engineering experience. I’ll ping you once I get things updated, which I’ll try to do sometime later today.Best,BrendanOn May 27, 2023, at 2:20 PM, Alon Roth @.***> wrote: @bpleahey I wonder what's the status of this PR? Its seems to me that this functionality is quite important and I wanted to see if someone is working on it before implementing it myself. I also wonder if you're making use of the exception in the next retry so it will be addressed by the llm?

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

[bpleahey]

@bpleahey I wonder what's the status of this PR?

Its seems to me that this functionality is quite important and I wanted to see if someone is working on it before implementing it myself.

I also wonder if you're making use of the exception in the next retry so it will be addressed by the llm?

Hey Alon, just following up. It looks like Harrison (or another contributor) has created general retry functionality when using openai's API here: langchain/llms/openai.py. I just hit a rate limit on OPENAI so I won't be able to do much today, but I just gave you access to the branch. You'll find a scratch folder with a notebook I was working on to catch errors. It's possible I missed some of the nuances with how the retries work, as I'm a relatively inexperienced software engineer, but I tried to provide some helpful info you can start working with.

Here is the list of TODOS I had:

• Restructure code to mirror new implementation for error handling: basic errors are now being thrown in langchain/chains/base.py:134. For consistency's sake, implementing a similar retry structure, or even overwriting the method could be a better approach then looping as I was doing. Particularly, the stop/wait implementation there seems cool, and is something I couldn't have come up with, especially not 5 months ago.
• Generate a more exhaustive list of SQL errors to be caught. I was attempting to do this using the notebook I created, and once I'm back under my rate limit am happy to keep helping with. SQLAlchemy is the library used in the SQL chain, and I included a link to the documentation in the base file (https://docs.sqlalchemy.org/en/20/core/exceptions.html).
• Most importantly, catching SQL queries that produce logical errors not caught at runtime. I'm happy to help with this part as well, depending on how intense my internship gets, as it seems to be a cool challenge depending on how much time I've got with work this summer.
• Fix minor bug with get_verbosity: get_verbosity in the base of the general chain was producing an error on line 25, saying langchain.verbose was not defined. I did a hacky fix of this by manually setting the field, and later reverted. It still seemed to be working in latest push so this might not really be an issue.

Best of luck with everything and happy to provide support—I like to finish things I've started, even if late.

Brendan

[alonroth]

@bpleahey Thanks for the clear instructions! I got the edit access and I'll try to give it some time later this week. Let's keep each other posted here for any updates or progress

Good luck with your internship!