dspy
dspy copied to clipboard
stanfordnlp
ValueError: ('Too many retries trying to get the correct output format. Try simplifying the requirements.', {'proposed_signatures': "ValueError('json output should start and end with { and }')"})
from pydantic import BaseModel, Field class Input(BaseModel): speech: str = Field(description="speech text")
class Output(BaseModel):
extracted_words: list[str] = Field(description="a list of extracted words from speech")
label: int = Field(ge=0, le=2, description="a signle integer--> 0,1 or 2")
class extract_words_and_label_signature(dspy.Signature): """take this speech text and output extracted_words and label"""
input: Input = dspy.InputField()
output: Output = dspy.OutputField()
def validate_Signature(example, pred, trace=None):
if type(pred.label) != type(example.label):
return False
else:
return False
import dspy from dspy.evaluate import Evaluate from dspy.teleprompt.signature_opt_typed import optimize_signature
evaluator = Evaluate(devset=converted_dataset, metric=validate_Signature, num_threads=1, display_progress=True)
result = optimize_signature( student=dspy.TypedPredictor(extract_words_and_label_signature), evaluator=evaluator, initial_prompts=20, n_iterations=100, # Increased number of iterations max_examples=30, verbose=True, prompt_model=gpt2, # Set do_sample=True here )
Setting pad_token_id to eos_token_id:50256 for open-end generation.
Found 1 typed predictors to optimize.
Generating 20 initial signatures for base...
Setting pad_token_id to eos_token_id:50256 for open-end generation.
Setting pad_token_id to eos_token_id:50256 for open-end generation.
Setting pad_token_id to eos_token_id:50256 for open-end generation.
Setting pad_token_id to eos_token_id:50256 for open-end generation.
ValueError Traceback (most recent call last)
2 frames /usr/local/lib/python3.10/dist-packages/dspy/functional/functional.py in forward(self, **kwargs) 233 {key: [r[key] for r in parsed_results] for key in signature.output_fields}, 234 ) --> 235 raise ValueError( 236 "Too many retries trying to get the correct output format. " + "Try simplifying the requirements.", 237 errors,
ValueError: ('Too many retries trying to get the correct output format. Try simplifying the requirements.', {'proposed_signatures': "ValueError('json output should start and end with { and }')"})
................... converted_dataset[0] Example({'speech': 'the uk has threatened to return radioactive waste to the eu if an agreement cannot be reached can not we keep the radioactive waste and send back all the paki migrants instead', 'extracted_tokens': ['send', 'back', 'all', 'the', 'paki', 'migrants', 'instead'], 'label': 0}) (input_keys={'speech'}).
let me know how to resolve this issue?
What is the meaning of the label 0, 1 or 2 and which text should be extracted?
If the model is confused by the task at hand, it might be unable to parse the output. What has helped me is adding a desc to the OutputField of the signature, e.g. output: Output = dspy.OutputField(desc=“Output from the speech text in the form of extracted hatespeech words and a label on whether the speech is hateful or not.”), and/or elaborating on the meaning of the extracted_words and label in the Output class.
hi @anushka192001
Running into the same issue and searching for answers i stumbled over this thread.
As I was able to find the root cause I would like to share:
The issue for me was the limiting default setting for max_tokens when instantiating the teacher_lm using dspy.OpenAI.
This leads to a cut off of a json-result internally in function _unwrap_json (See code https://github.com/stanfordnlp/dspy/blob/main/dspy/functional/functional.py#L408).
This did not work and lead to the same error as you mentioned:
teacher_lm = dspy.OpenAI(model="gpt-4-turbo-2024-04-09")
This got rid of the error message:
teacher_lm = dspy.OpenAI(model="gpt-4-turbo-2024-04-09", max_tokens=1000)
@anushka192001 what is teacher_lm in my code would be gpt2 in your code.
@anushka192001 I just ran into a second issue when using 'mixtral'. After looking closely on the line number of your error-message i realised that I am now hitting this and it was a different one before.
For me now the error on line "235" went away when I used GPT-4. I was trying with 'mixtral' and that one did not comply to these strict json-parsing rules.
Hope that clears your path
is there any good open source model which doesn't gives much error with dspy?
Erm,
Code has been redacted
In the UK that p-word is very very racist. Incredibly so. Maybe @anushka192001 you can reproduce without the racism?
I suspect the root-issue for this and https://github.com/stanfordnlp/dspy/issues/875 is that you're running into the content filters for the models you're using.
