llama_index Improve text-to-sql performance with updated prompt and stop token

Improve text-to-sql performance with updated prompt and stop token

Open ekzhu opened this issue 2 years ago • 1 comments

On spider benchmark dev set, execution accuracy increased to 70% from 50% (on a 1% sample queries).
Added stop_token to Prompt class to allow new prompts to specify when to ask LLM to stop generation. Particularity useful when you ask LLM to generate items 1, 2, 3 but only want item 1, and items 2 and 3 are just there due to prompt engineering.
Updated text-to-sql prompt which utilizes stop_token.
Use ChatGPT instead of davinci for running benchmark to save $$$
Add foreign keys to table context in structured index.

Mar 14 '23 07:03 ekzhu

@ekzhu quick question: how does the performance of ChatGPT compare to davinci in your experience?

Mar 14 '23 15:03 Disiok

@ekzhu quick question: how does the performance of ChatGPT compare to davinci in your experience?

Let's test it

Mar 16 '23 04:03 ekzhu