DSPy.Suggest halts execution instead of continuing

[bllchmbrs]

trafficstars

Reproduction

import dspy

class Repro(dspy.Signature):
    """Product some text that includes numbers. Be sure to include numbers from 1 to 10."""

    input = dspy.InputField()
    body = dspy.OutputField(format=str)
    
def contains_a_number(output):
    contains = []
    for x in range(0,10,2):
        if str(x) in output:
            contains.append(str(x))

    return contains

class SectionWriter(dspy.Module):
    def __init__(self):
        super().__init__()
        self.prog = dspy.ChainOfThought(Repro)

    def forward(self, input):
        result = self.prog(input=input)
        body = result.body

        nums = contains_a_number(body)
        message = "Avoid these numbers: " + ", ".join(
            nums
        )
        as_bool = len(nums) <= 1

        dspy.Suggest(as_bool, message)
        return dspy.Prediction(body=body)

gpt3 = dspy.OpenAI(
    model="gpt-3.5-turbo",
    max_tokens=4000,
    model_type="chat",
    api_base="http://0.0.0.0:4000/",
)
with dspy.context(lm=gpt3):
    SectionWriter().forward(input="something")

Result

ERROR:dspy.primitives.assertions:2024-05-04T04:17:51.517685Z [error    ] SuggestionFailed: Avoid these numbers: 0, 2, 4, 6, 8 [dspy.primitives.assertions] filename=assertions.py lineno=111

    111         dspy.logger.error(f"SuggestionFailed: {self.msg}")
--> 112         raise DSPySuggestionError(
    113             id=self.id,
    114             msg=self.msg,
    115             target_module=self.target_module,
    116             state=dsp.settings.trace,
    117             is_metric=self.is_metric,
    118         )
    119 else:
    120     raise ValueError("Suggestion function should always return [bool]")

DSPySuggestionError: Avoid these numbers: 0, 2, 4, 6, 8

Expectation

That it continues executing. I expect it to (a) find the numbers and (b) complain about them. But shouldn't it just continue executing? I get the same error if I am trying to run an optimization as well.

Thoughts

I think there might be a bug in the implementation of Suggestions but I'm not sure.

I followed the docs quite closely and have the simplest example - shouldn't this just keep executing?

[arnavsinghvi11]

Hi @bllchmbrs , thanks for raising this. Activating DSPy Assertions/Suggestions actually requires wrapping the SectionWriter class with the backtracking/Retry logic. This will ensure that when the Suggestion is flagged with a failure in the validation function, this will trigger the corresponding Retry logic.

Feel free to reference this part of the documentation and let me know if this helps!

from dspy.primitives.assertions import assert_transform_module, backtrack_handler

baleen_with_assertions = assert_transform_module(SimplifiedBaleenAssertions(), backtrack_handler)

# backtrack_handler is parameterized over a few settings for the backtracking mechanism
# To change the number of max retry attempts, you can do
baleen_with_assertions_retry_once = assert_transform_module(SimplifiedBaleenAssertions(), 
    functools.partial(backtrack_handler, max_backtracks=1))

Alternatively, you can also directly call activate_assertions on the program with dspy.Assert/Suggest statements using the default backtracking mechanism (max_backtracks=2):

baleen_with_assertions = SimplifiedBaleenAssertions().activate_assertions()

In your case, you could do:

from dspy.primitives.assertions import assert_transform_module, backtrack_handler 

section_writer_with_assertions = assert_transform_module(SectionWriter(), backtrack_handler)
section_writer_with_assertions(input="something"))

There also some relevant Assertions FAQs that you may find relevant (and that I will update in the documentation soon to make it more clear!).

[bllchmbrs]

Thank you! Will give this a try and will open a PR for the docs too. Appreciate your help!

On Sat, May 4, 2024 at 15:20 arnavsinghvi11 @.***> wrote:

Hi @bllchmbrs https://github.com/bllchmbrs , thanks for raising this. Activating DSPy Assertions/Suggestions actually requires wrapping the SectionWriter class with the backtracking/Retry logic. This will ensure that when the Suggestion is flagged with a failure in the validation function, this will trigger the corresponding Retry logic.

Feel free to reference this part of the documentation and let me know if this helps!

from dspy.primitives.assertions import assert_transform_module, backtrack_handler

baleen_with_assertions = assert_transform_module(SimplifiedBaleenAssertions(), backtrack_handler)

backtrack_handler is parameterized over a few settings for the backtracking mechanism

To change the number of max retry attempts, you can do

baleen_with_assertions_retry_once = assert_transform_module(SimplifiedBaleenAssertions(), functools.partial(backtrack_handler, max_backtracks=1))

Alternatively, you can also directly call activate_assertions on the program with dspy.Assert/Suggest statements using the default backtracking mechanism (max_backtracks=2):

baleen_with_assertions = SimplifiedBaleenAssertions().activate_assertions()

In your case, you could do:

from dspy.primitives.assertions import assert_transform_module, backtrack_handler

section_writer_with_assertions = assert_transform_module(SectionWriter(), backtrack_handler) section_writer_with_assertions(input="something"))

There also some relevant Assertions FAQs https://github.com/stanfordnlp/dspy/issues/434 that you may find relevant (and that I will update in the documentation soon to make it more clear!).

— Reply to this email directly, view it on GitHub https://github.com/stanfordnlp/dspy/issues/965#issuecomment-2094462501, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMRAB3ODZ7TNQMIUUUCTC3ZAVNJHAVCNFSM6AAAAABHGP5NEWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOJUGQ3DENJQGE . You are receiving this because you were mentioned.Message ID: @.***>

[bllchmbrs]

Ok, this is much clearer to me.

As I mentioned, I am happy to help out with the docs. @arnavsinghvi11 , may I make a proposal here (then subsequently implement that proposal) that includes some of the suggestions in #434 and an overall restructuring?

I think that Assertions/Suggestions are an exciting aspect of DSPy and want as many people as possible to understand and use them.

[lebsral]

This has been a very helpful thread. It made several things click for me.

I'm still a little confused with the functools.partial

I'm probably not importing something correctly.

[bllchmbrs]

The functools partial just changes the backtracker function so that you can modify the number of retries. That's just a python thing. You don't need to set it otherwise.

@lebsral

[okhat]

Hey all, is this complete or pending something?

[bllchmbrs]

This is complete

On Mon, May 20, 2024 at 03:23 Omar Khattab @.***> wrote:

Hey all, is this complete or pending something?

— Reply to this email directly, view it on GitHub https://github.com/stanfordnlp/dspy/issues/965#issuecomment-2120149078, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMRAB3BPWGTE4GDOYJVM5LZDHFKFAVCNFSM6AAAAABHGP5NEWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRQGE2DSMBXHA . You are receiving this because you were mentioned.Message ID: @.***>

[Celtico01]

Here: baleen_with_assertions = SimplifiedBaleenAssertions().activate_assertions(), can I pass the max_backtracks=n like that: SimplifiedBaleenAssertions().activate_assertions(max_backtracks=n)?