After saving and loading, the program optimized by BayesianSignatureOptimizer encounters an error when calling forward

[lzjever]

code as follows:

def train():
    teleprompter = BayesianSignatureOptimizer(prompt_model=sql_llm, task_model=sql_llm, metric=assess_api_selector, n=10, init_temperature=0.9, verbose=True,track_stats=True)
    evaluate_api_selector = Evaluate(devset=dataset_api_calling_gt_sample[10:20], metric=assess_api_selector, num_threads=1, display_progress=True, display_table=0)


    kwargs = dict(num_threads=1, display_progress=True, display_table=0)
    uncompiled_prompt = APISelector()
    #evaluate_api_selector(uncompiled_prompt)
    compiled_prompt_opt = teleprompter.compile(uncompiled_prompt, devset=dataset_api_calling_gt_sample[:10], optuna_trials_num=40, max_bootstrapped_demos=3, max_labeled_demos=5, eval_kwargs=kwargs)
    compiled_prompt_opt.save('0221_2_api_selector_expert_trainer.json')

def test():
    dataset_api_calling_gt_sample = [dspy.Example(question=x["question"],selected_api=x["api"]).with_inputs('question') for ind, x in result.iterrows()]
    compiled_prompt_opt = APISelector()
    compiled_prompt_opt.load('0221_2_api_selector_expert_trainer.json')
    print(compiled_prompt_opt)
    #compiled_prompt_opt.forward('What's the time?')
    evaluate_api_selector = Evaluate(devset=dataset_api_calling_gt_sample[20:30] , metric=assess_api_selector, num_threads=1, display_progress=True, display_table=0)
    evaluate_api_selector(compiled_prompt_opt)


train()
test()

The error occurs here:

Traceback (most recent call last):
  File "/home/percy/works/hongyou/walagi-offline/api_selector_expert_trainer.py", line 134, in <module>
    test()
  File "/home/percy/works/hongyou/walagi-offline/api_selector_expert_trainer.py", line 126, in test
    compiled_prompt_opt.forward('What's the time?')
  File "/home/percy/works/hongyou/walagi-offline/api_selector_expert_trainer.py", line 78, in forward
    prediction = self.generate_answer(question=question,api_inventory = self.api_inventory_text)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/percy/works/hongyou/walagi/venv/lib/python3.11/site-packages/dspy/predict/predict.py", line 72, in __call__
    return self.forward(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/percy/works/hongyou/walagi/venv/lib/python3.11/site-packages/dspy/predict/chain_of_thought.py", line 66, in forward
    return super().forward(signature=signature, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/percy/works/hongyou/walagi/venv/lib/python3.11/site-packages/dspy/predict/predict.py", line 104, in forward
    x, C = dsp.generate(signature, **config)(x, stage=self.stage)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/percy/works/hongyou/walagi/venv/lib/python3.11/site-packages/dsp/primitives/predict.py", line 77, in do_generate
    prompt = template(example)
             ^^^^^^^^^^^^^^^^^
  File "/home/percy/works/hongyou/walagi/venv/lib/python3.11/site-packages/dsp/templates/template_v2.py", line 218, in __call__
    ademos = [
             ^
  File "/home/percy/works/hongyou/walagi/venv/lib/python3.11/site-packages/dsp/templates/template_v2.py", line 219, in <listcomp>
    self.query(demo, is_demo=True)
  File "/home/percy/works/hongyou/walagi/venv/lib/python3.11/site-packages/dsp/templates/template_v2.py", line 105, in query
    if self._has_augmented_guidelines() and ("augmented" in example and example.augmented):
                                                                        ^^^^^^^^^^^^^^^^^
AttributeError: 'dict' object has no attribute 'augmented'

Fixed this by replacing all xx.augmented to xx['augmented'] in the template_v2.py. I haven't gone through all the code yet, so I'm not sure if this modification will have any side effects.

[lzjever]

Hello, everyone. I've gone through all the posts here and it seems that no one else has encountered this particular issue. I'm wondering if perhaps I'm not using it correctly.

Here's what I'm trying to do: I want to save an optimized program to JSON and then reuse it in another process later on. My code looks like this:

# Save the optimized program to JSON, and reuse it in another process later on:
compiled_prompt_opt.save('0221_2_api_selector_expert_trainer.json')

# In another process:
compiled_prompt_opt.load('0221_2_api_selector_expert_trainer.json')
compiled_prompt_opt.forward("What's the time?")

However, this always results in an error: AttributeError: 'dict' object has no attribute 'augmented'.

I tried changing the optimizer from BayesianSignatureOptimizer to BootstrapFewShotWithRandomSearch, but the error persists. A temporary workaround I found is to replace all instances of xx.augmented with xx['augmented'] in the template_v2.py, which seems to fix the issue.

I'm not sure if this is a bug or if I'm making a mistake in my code.

[insop]

I think I see the same issue.

My code is something like this

compiled_rag.load(f"compiled_rag.json")

And I got this error, and my branch is up-to-date (42a5943379d28d1673dc8fe332a3d596efdfc7a3) , I had to since save was failing before this issue.

File ~/Projects/Github/NLP/dspy/dsp/templates/template_v2.py:210, in <listcomp>(.0)
    203 if self.fields[-1].input_variable in example:
    204     del example[self.fields[-1].input_variable]
    206 rdemos = [
    207     self.query(demo, is_demo=True)
    208     for demo in example.demos
    209     if (
--> 210         ("augmented" not in demo or not demo.augmented)
    211         and (  # validate that the training example has the same primitive input var as the template
    212             self.fields[-1].input_variable in demo
    213             and demo[self.fields[-1].input_variable] is not None
    214         )
AttributeError: 'dict' object has no attribute 'augmented'

[shacker27]

As an additional piece of information, I just hit the same issue. I noticed that poetry upgraded our dspy-ai version from 2.1.10 to 2.3.1. Reverting that change and pinning to 2.1.10 has things working again while loading the JSON file from disk, so it seems like something that has only been introduced in the past two weeks.

[drawal1]

I am running into the same error after upgrading to 2.3.4. ("augmented" not in demo or not demo.augmented) probably needs to be changed to (not demo.get('augmented', False))

[okhat]

@drawal1 good catch, do you want to make this a PR?