AutoPrompt icon indicating copy to clipboard operation
AutoPrompt copied to clipboard

Calibrated prompt generated is completely different from initial prompt.

Open shobhnadhami opened this issue 10 months ago • 3 comments

I have a prompt which is used to generate sql query from the input text given by a user. I am trying to optimize prompt using run_generation_pipeline.py, but I am getting completely different Calibrated prompt. Below are the inputs provided:

--task_description:
Assistant is a large language model that is tasked to generate SQL query based on details and examples provided in prompt.

--prompt: 
We have 2 tables:
    Employee: Employee table have information regarding all the employees in a company.
    Below are the attributes of Employee table
        empid: empid column contains employee id. empid is a primary key of Employee table.
        name: Name column contains name of the employee
        salary: salary column contains salary of the employee
        department_id: department_id column contains employee's department id. It is a foreign key from Department table.
    Department: Department table have information regarding all the department of a company.
    Below are the attributes of Department table
        department_id: department_id contains the id of the department. department_id is primary key of Department table.
        department_name: department_name contains name of the department.
***Below are few examples***:
##Example 1
user query: what is empid of employees in department A?
output: Select Employee.empid
        From Employee 
        Join Department 
        on Employee.department_id = Department.department_id
        Where Department.department_id = 'A';
##Example 2
user query: what is salary of employee with empid=1?
output: Select salary
        From Employee 
        Where empid = 1;
** End of Examples **

Your task is to generate SQL query from natural language input provided by user.
Your task is to understand natural language input and provide SQL query to fetch information asked in natural language input from above tables.

annotator instruction in config_default.yml:
        instruction:
            'We have two tables Employee and Department.
            Employee table have empid, name, salary, department_id as columns
            Department table have department_id, department_name as columns
            You will be given a query in natural language and its interpreted sql query to fetch data from above table. 
            Asses interpreted SQL query with respect to natural language input and table provided. Answer 1 if SQL query is relevant 
           and correct otherwise 0.'

output given by AutoPrompt:

Calibrated prompt score: 1.0
Calibrated prompt: Your task is to generate accurate and context-specific SQL queries based on natural language input provided by the user. Please include specific examples of natural language input and the corresponding expected SQL queries. Additionally, describe the database schema and table structure to provide more context for query generation. Aim for a higher score by improving the model's understanding and accuracy in generating SQL queries. 

Output given is not relevant to the task. Am I providing the wrong inputs or missing some inputs that needs to be provided?

shobhnadhami avatar Apr 03 '24 14:04 shobhnadhami