dspy
dspy copied to clipboard
Published
20 hours ago •
stanfordnlp
stanfordnlp
'Predict' object has no attribute 'new_signature'.
I'm getting this error when running MIPRO:
AttributeError: 'Predict' object has no attribute 'new_signature'. Did you mean: 'signature'?
This is how I run MIPRO:
teleprompter = dspy.teleprompt.MIPRO(metric = great_tweet_metric, verbose=True)
kwargs = dict(num_threads=16, display_progress=True, display_table=0)
tweet_generator_compiled = teleprompter.compile(tweet_generator, trainset=trainset, num_trials=2, max_bootstrapped_demos=4, max_labeled_demos=5, eval_kwargs=kwargs, requires_permission_to_run=False)
More output:
0%| | 0/50 [00:00<?, ?it/s]2024-06-06T01:00:54.964938Z [error ] Failed to run or to evaluate example Example({'reference_text': '---\nsidebar_position: 2\n---\n\n# retrieve.ChromadbRM\n\n### Constructor\n\nInitialize an instance of the `ChromadbRM` class, with the option to use OpenAI\'s embeddings or any alternative supported by chromadb, as detailed in the official [chromadb embeddings documentation](https://docs.trychroma.com/embeddings).\n\n```python\nChromadbRM(\n collection_name: str,\n persist_directory: str,\n embedding_function: Optional[EmbeddingFunction[Embeddable]] = OpenAIEmbeddingFunction(),\n k: int = 7,\n)\n```\n\n**Parameters:**\n- `collection_name` (_str_): The name of the chromadb collection.\n- `persist_directory` (_str_): Path to the directory where chromadb data is persisted.\n- `embedding_function` (_Optional[EmbeddingFunction[Embeddable]]_, _optional_): The function used for embedding documents and queries. Defaults to `DefaultEmbeddingFunction()` if not specified.\n- `k` (_int_, _optional_): The number of top passages to retrieve. Defaults to 7.\n\n### Methods\n\n#### `forward(self, query_or_queries: Union[str, List[str]], k: Optional[int] = None) -> dspy.Prediction`\n\nSearch the chromadb collection for the top `k` passages matching the given query or queries, using embeddings generated via the specified `embedding_function`.\n\n**Parameters:**\n- `query_or_queries` (_Union[str, List[str]]_): The query or list of queries to search for.\n- `k` (_Optional[int]_, _optional_): The number of results to retrieve. If not specified, defaults to the value set during initialization.\n\n**Returns:**\n- `dspy.Prediction`: Contains the retrieved passages, each represented as a `dotdict` with schema `[{"id": str, "score": float, "long_text": str, "metadatas": dict }]`\n\n### Quickstart with OpenAI Embeddings\n\nChromadbRM have the flexibility from a variety of embedding functions as outlined in the [chromadb embeddings documentation](https://docs.trychroma.com/embeddings). While different options are available, this example demonstrates how to utilize OpenAI embeddings specifically.\n\n```python\nfrom dspy.retrieve.chromadb_rm import ChromadbRM\nimport os\nimport openai\nfrom chromadb.utils.embedding_functions import OpenAIEmbeddingFunction\n\nembedding_function = OpenAIEmbeddingFunction(\n api_key=os.environ.get(\'OPENAI_API_KEY\'),\n model_name="text-embedding-ada-002"\n)\n\nretriever_model = ChromadbRM(\n \'your_collection_name\',\n \'/path/to/your/db\',\n embedding_function=embedding_function,\n k=5\n)\n\nresults = retriever_model("Explore the significance of quantum computing", k=5)\n\nfor result in results:\n print("Document:", result.long_text, "\\n")\n```\n'}) (input_keys={'reference_text'}) with <function great_tweet_metric at 0x7f2d108eb010> due to 'Predict' object has no attribute 'new_signature'. [dspy.teleprompt.bootstrap] filename=bootstrap.py lineno=204
2024-06-06T01:00:54.969715Z [error ] Failed to run or to evaluate example Example({'reference_text': '---\nsidebar_position: \n---\n\n# dspy.Snowflake\n\n### Usage\n\n```python\nimport dspy\nimport os\n\nconnection_parameters = {\n\n "account": os.getenv(\'SNOWFLAKE_ACCOUNT\'),\n "user": os.getenv(\'SNOWFLAKE_USER\'),\n "password": os.getenv(\'SNOWFLAKE_PASSWORD\'),\n "role": os.getenv(\'SNOWFLAKE_ROLE\'),\n "warehouse": os.getenv(\'SNOWFLAKE_WAREHOUSE\'),\n "database": os.getenv(\'SNOWFLAKE_DATABASE\'),\n "schema": os.getenv(\'SNOWFLAKE_SCHEMA\')}\n\nlm = dspy.Snowflake(model="mixtral-8x7b",credentials=connection_parameters)\n```\n\n### Constructor\n\nThe constructor inherits from the base class `LM` and verifies the `credentials` for using Snowflake API.\n\n```python\nclass Snowflake(LM):\n def __init__(\n self, \n model,\n credentials,\n **kwargs):\n```\n\n**Parameters:**\n- `model` (_str_): model hosted by [Snowflake Cortex](https://docs.snowflake.com/en/user-guide/snowflake-cortex/llm-functions#availability).\n- `credentials` (_dict_): connection parameters required to initialize a [snowflake snowpark session](https://docs.snowflake.com/en/developer-guide/snowpark/reference/python/latest/api/snowflake.snowpark.Session)\n\n### Methods\n\nRefer to [`dspy.Snowflake`](https://dspy-docs.vercel.app/api/language_model_clients/Snowflake) documentation.\n'}) (input_keys={'reference_text'}) with <function great_tweet_metric at 0x7f2d108eb010> due to 'Predict' object has no attribute 'new_signature'. [dspy.teleprompt.bootstrap] filename=bootstrap.py lineno=2042024-06-06T01:00:54.990462Z [error ] Failed to run or to evaluate example Example({'reference_text': '---\nsidebar_position: 3\n---\n\n# teleprompt.Ensemble\n\n### Constructor\n\nThe constructor initializes the `Ensemble` class and sets up its attributes. This teleprompter is designed to create ensembled versions of multiple programs, reducing various outputs from different programs into a single output.\n\n```python\nclass Ensemble(Teleprompter):\n def __init__(self, *, reduce_fn=None, size=None, deterministic=False):\n```\n\n**Parameters:**\n- `reduce_fn` (_callable_, _optional_): Function used to reduce multiple outputs from different programs into a single output. A common choice is `dspy.majority`. Defaults to `None`.\n- `size` (_int_, _optional_): Number of programs to randomly select for ensembling. If not specified, all programs will be used. Defaults to `None`.\n- `deterministic` (_bool_, _optional_): Specifies whether ensemble should operate deterministically. Currently, setting this to `True` will raise an error as this feature is pending implementation. Defaults to `False`.\n\n### Method\n\n#### `compile(self, programs)`\n\nThis method compiles an ensemble of programs into a single program that when run, can either randomly sample a subset of the given programs to produce outputs or use all of them. The multiple outputs can then be reduced into a single output using the `reduce_fn`.\n\n**Parameters:**\n- `programs` (_list_): List of programs to be ensembled.\n\n**Returns:**\n- `EnsembledProgram` (_Module_): An ensembled version of the input programs.\n\n### Example\n\n```python\nimport dspy\nfrom dspy.teleprompt import Ensemble\n\n# Assume a list of programs\nprograms = [program1, program2, program3, ...]\n\n# Define Ensemble teleprompter\nteleprompter = Ensemble(reduce_fn=dspy.majority, size=2)\n\n# Compile to get the EnsembledProgram\nensembled_program = teleprompter.compile(programs)\n```'}) (input_keys={'reference_text'}) with <function great_tweet_metric at 0x7f2d108eb010> due to 'Predict' object has no attribute 'new_signature'. [dspy.teleprompt.bootstrap] filename=bootstrap.py lineno=204
2024-06-06T01:00:54.995006Z [error ] Failed to run or to evaluate example Example({'reference_text': '---\nsidebar_position: 5\n---\n\n# dspy.HFClientVLLM\n\n### Usage\n\n```python\nlm = dspy.HFClientVLLM(model="meta-llama/Llama-2-7b-hf", port=8080, url="http://localhost")\n```\n\n### Prerequisites\n\nRefer to the [vLLM Server](https://dspy-docs.vercel.app/api/language_model_clients/HFClientVLLM) section of the `Using Local Models` documentation.\n\n### Constructor\n\nRefer to [`dspy.TGI`](https://dspy-docs.vercel.app/api/language_model_clients/TGI) documentation. Replace with `HFClientVLLM`.\n\n### Methods\n\nRefer to [`dspy.OpenAI`](https://dspy-docs.vercel.app/api/language_model_clients/OpenAI) documentation.'}) (input_keys={'reference_text'}) with <function great_tweet_metric at 0x7f2d108eb010> due to 'Predict' object has no attribute 'new_signature'. [dspy.teleprompt.bootstrap] filename=bootstrap.py lineno=204
14%|█████ | 7/50 [00:00<00:00, 147.64it/s]
Traceback (most recent call last):
File "/home/tom/dspy/dspy/primitives/assertions.py", line 220, in wrapper
result = func(*args, **kwargs)
File "/home/tom/tweet_generation/./main.py", line 127, in forward
run_all_checks(tweet_text)
File "/home/tom/tweet_generation/./main.py", line 110, in run_all_checks
suggest_check(generated_text, "Does the assessed tweet sound natural? Say no if it sounds like an ad.", "The tweet does not sound natural. Please revise for quality.")
File "/home/tom/tweet_generation/./main.py", line 100, in suggest_check
dspy.Suggest(assessment_bool, suggestion)
File "/home/tom/dspy/dspy/primitives/assertions.py", line 74, in __init__
self.__call__()
File "/home/tom/dspy/dspy/primitives/assertions.py", line 112, in __call__
raise DSPySuggestionError(
dspy.primitives.assertions.DSPySuggestionError: The tweet does not sound natural. Please revise for quality.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/tom/tweet_generation/./main.py", line 189, in <module>
tweet_generator_compiled = teleprompter.compile(tweet_generator, trainset=trainset, num_trials=2, max_bootstrapped_demos=4, max_labeled_demos=5, eval_kwargs=kwargs, requires_permission_to_run=False)
File "/home/tom/dspy/dspy/teleprompt/mipro_optimizer.py", line 452, in compile
candidate_program = tp.compile(student=module.deepcopy(), trainset=shuffled_trainset)
File "/home/tom/dspy/dspy/teleprompt/bootstrap.py", line 84, in compile
self._bootstrap()
File "/home/tom/dspy/dspy/teleprompt/bootstrap.py", line 147, in _bootstrap
success = self._bootstrap_one_example(example, round_idx)
File "/home/tom/dspy/dspy/teleprompt/bootstrap.py", line 203, in _bootstrap_one_example
raise e
File "/home/tom/dspy/dspy/teleprompt/bootstrap.py", line 183, in _bootstrap_one_example
prediction = teacher(**example.inputs())
File "/home/tom/dspy/dspy/primitives/program.py", line 26, in __call__
return self.forward(*args, **kwargs)
File "/home/tom/dspy/dspy/primitives/assertions.py", line 294, in forward
return wrapped_forward(*args, **kwargs)
File "/home/tom/dspy/dspy/primitives/assertions.py", line 260, in wrapper
output_fields = error_state[0].new_signature.output_fields
AttributeError: 'Predict' object has no attribute 'new_signature'. Did you mean: 'signature'?
Ah okay. Would it make sense to throw that as the error message? I spent quite a bit of time trying to solve this myself
Feel free to add it @tom-doerr ! Assertions are currently compatible with the FewShot optimizers, and we'll eventually get them integrated with the new Signature optimizers coming out.
MIPRO and assertions not compatible currently unfortunately
I encountered the same error when using BootstrapFewShotWithRandomSearch. Does BootstrapFewShotWithRandomSearch also not support assertions?
Hey, I'm also seeing this with BootstrapFewShotWithRandomSearch and Suggestion.
Here's the error:
2024-09-09 01:34:17,991 - dspy.primitives.assertions - INFO - [2m2024-09-09T05:34:17.990976Z[0m [[32m[1minfo [0m] [1mSuggestionFailed: The quizbowl question does not contain the exact, specified number of sentences/clues. Please rewrite it such that it has exactly the specified number (num_clues).[0m [[0m[1m[34mdspy.primitives.assertions[0m][0m [36mfilename[0m=[35massertions.py[0m [36mlineno[0m=[35m108[0m
Traceback (most recent call last):
File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/primitives/assertions.py", line 220, in wrapper
result = func(*args, **kwargs)
File "/some/dir/some/dir/question_generation/pyramodelQBGeneratorDSPy.py", line 234, in forward
dspy.Suggest(
File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/primitives/assertions.py", line 74, in __init__
self.__call__()
File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/primitives/assertions.py", line 112, in __call__
raise DSPySuggestionError(
dspy.primitives.assertions.DSPySuggestionError: The quizbowl question contains facts that are not grounded in the context of the Wikipedia sentences it was given. Please revise for accuracy.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/some/dir/some/dir/question_generation/pyramodelDSPyTrainer.py", line 114, in <module>
main(args)
File "/some/dir/some/dir/question_generation/pyramodelDSPyTrainer.py", line 68, in main
optimized_program = teleprompter.compile(student=pyraGenerator, teacher=pyraGenerator, trainset=train_dataset, valset=val_dataset)
File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/teleprompt/random_search.py", line 123, in compile
score, subscores = evaluate(program2, return_all_scores=True)
File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/evaluate/evaluate.py", line 193, in __call__
reordered_devset, ncorrect, ntotal = self._execute_multi_thread(
File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/evaluate/evaluate.py", line 110, in _execute_multi_thread
example_idx, example, prediction, score = future.result()
File "/some/dir/miniconda3/envs/qa5/lib/python3.10/concurrent/futures/_base.py", line 451, in result
return self.__get_result()
File "/some/dir/miniconda3/envs/qa5/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/some/dir/miniconda3/envs/qa5/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/evaluate/evaluate.py", line 103, in cancellable_wrapped_program
return wrapped_program(idx, arg)
File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/evaluate/evaluate.py", line 160, in wrapped_program
prediction = program(**example.inputs())
File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/primitives/program.py", line 26, in __call__
return self.forward(*args, **kwargs)
File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/primitives/assertions.py", line 294, in forward
return wrapped_forward(*args, **kwargs)
File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/primitives/assertions.py", line 260, in wrapper
output_fields = error_state[0].new_signature.output_fields
AttributeError: 'Predict' object has no attribute 'new_signature'. Did you mean: 'signature'?
CRITICAL: Exiting due to uncaught exception AttributeError: 'Predict' object has no attribute 'new_signature'
Here's the relevant code:
training script
pyraGenerator = assert_transform_module(PyramodelQbQuestionGeneratorDSPy().map_named_predictors(Retry), backtrack_handler)
pyra_df = pd.read_csv(filepath_or_buffer=DATA_DIR / f"dspy_training/pyra_questions.csv")
pyra_dataset = dl.from_pandas(
pyra_df,
fields=("question", "clues", "wiki_sents", "num_clues", "wiki_page_title", "max_tokens"),
input_keys=("clues", "wiki_sents", "num_clues", "wiki_page_title", "max_tokens")
)
splits = dl.train_test_split(pyra_dataset, train_size=0.5)
train_dataset = splits['train']
val_dataset = splits['test']
config = dict(max_bootstrapped_demos=16, max_labeled_demos=16, num_candidate_programs=16, num_threads=16)
teleprompter = BootstrapFewShotWithRandomSearch(metric=composite_qb_metric, **config)
optimized_program = teleprompter.compile(student=pyraGenerator, teacher=pyraGenerator, trainset=train_dataset, valset=val_dataset)
optimized_program.save(DATA_DIR / f"dspy_training/checkpoint.json", save_field_meta=True)
generator
class AssessQBQuestion(dspy.Signature):
"""Assess the quality of a generated quizbowl question along the specified dimension."""
assessment_question = dspy.InputField(desc="Question used to assess the quality of given quizbowl question.")
assessed_quizbowl_question = dspy.InputField(desc="Quizbowl question being assessed")
wiki_page_title = dspy.InputField(desc="Name of relevant Wikipedia page about the answer")
num_clues = dspy.InputField(desc="Number of clues in the quizbowl question. Typically, 3-8. Ignore if N/A")
max_tokens = dspy.InputField(desc="Number of allowed tokens. Ignore if N/A.")
context = dspy.InputField(desc="Ignore if N/A.")
assessment_answer: str = dspy.OutputField(desc="Yes or No")
IS_FAITHFUL = \
("Is the assessed quizbowl question grounded in the context? Say no if "
"it includes significant facts not in the context.")
def is_faithful(gold, pred, trace=None) -> bool:
return is_assessment_yes(
dspy.Predict(AssessQBQuestion)(
assessed_quizbowl_question=pred.generated_question,
assessment_question=IS_FAITHFUL,
wiki_page_title=pred.wiki_page_title,
num_clues=NA, max_tokens=NA,
context=pred.wiki_sents
)
)
class CluesToQBQuestion(dspy.Signature):
"""Write the list of Wikipedia sentences into an ACF-style quizbowl question."""
wiki_page_title: str = dspy.InputField(desc="The title of a Wikipedia page about the answer of the quizbowl question.")
num_clues: str = dspy.InputField(desc="The number of clues (sentences) to be in the resulting quizbowl question.")
max_tokens: str = dspy.InputField(desc="The maximum allowed tokens.")
clues: str = dspy.InputField(desc="A list of quizbowl clues.")
quizbowl_question: str = dspy.OutputField(desc="The generated quizbowl question")
generator (forward)
def forward(
self,
wiki_page_title: str,
wiki_sents: List[str],
num_clues: str,
max_tokens: str,
clues: str=[] # Optional argument because it will be provided during training but not inference
) -> dspy.Prediction:
if type(num_clues) == int: num_clues = str(num_clues) # Typing with DSPy is not fun
if type(max_tokens) == int: max_tokens = str(max_tokens)
if not clues:
clues = []
for i, sent in enumerate(wiki_sents):
clue = \
self.clueMasker.mask_wiki_sent(
wiki_page=wiki_page_title,
wiki_sent=sent,
clue_pos=(1+i)/num_clues)
clues.append(clue)
keywords = guesser.get_topk_keywords(
label=wiki_page_title, wiki_sents=wiki_sents, k=10)
self.mask_keywords(wiki_page_title=wiki_page_title, wiki_sents=clues, mask_first_k=2, keywords=keywords)
generated_question = self.finishing_touches(wiki_page_title, num_clues, max_tokens, clues)
# The keywords property is used by a metric to make sure keywords have been removed
pred = dspy.Prediction(
generated_question=generated_question,
wiki_sents=wiki_sents,
num_clues=num_clues,
max_tokens=str(int(max_tokens) + random.randint(-20, 20)),
wiki_page_title=wiki_page_title,
page=wiki_page_title)
dspy.Suggest(
is_faithful(None, pred),
msg=("The quizbowl question contains facts that are not grounded "
"in the context of the Wikipedia sentences it was given. "
"Please revise for accuracy."),
# target_module=CluesToQBQuestion,
is_metric=True
)
I commented out # target_module=CluesToQBQuestion because I'm getting this issue
@tom-doerr Could you please elaborate on how you fixed this since it looks like you've fixed this problem (issue)?
I don't quite understand what is meant by "updated the signature class (target_module)".
Suggest takes the attribute target_module:
dspy.Suggest(is_assessment_yes(contact_person_assessment.assessment_answer), "The text contains names and titles not in the context. Please revise for accuracy.", target_module=GenerateMail)
I believe I renamed GenerateMail to something else. If I remember correctly I just updated the class name.
Interesting thank you @tom-doerr ! I'll take a look and update this issue if I find anything.
wait.. suggestions work with a target module configured? pls confirm
but: I remembered this issue, wanted to comment because I had a thought..: wouldn't wrapping the compile call in a with dspy.context(bypass_suggest=True, bypass_assert=True) block disable the assertions just for the compilation?
and if so, wouldn't that be ideal, as catching the errors with assertions might make the optimizer less effective / artificially elevate the score?
(haven't tested it yet, read through the Assert module source, saw those dspy.settings options, and thought of this solution on my way home, but I think it should work actually)
Would be great if this was handled better, just spent some time trying to debug this error. As I understand it this is also caused by MIPRO not supporting assertions.
ERROR dspy.primitives.assertions: Module not found in trace. If passing a DSPy Signature, please specify the intended module for the assertion (e.g., use `target_module = self.my_module(my_signature)` instead of `target_module = my_signature`).
Closing because none of these components (new_signature; old implementation of assertions v1) are in 2.6