DSPy like fewshot to have more control on the specific input data

[fivejjs]

DSPy is very powerful to "optimize" the fewshot cases: https://dspy-docs.vercel.app/docs/building-blocks/optimizers Can textgrad work with the fewshot with more context and input/output fields?

[mertyg]

Hey @fivejjs !! Thank you so much for the interest. I agree -- DSPy is indeed very powerful for constructing few-shot examples.

Can you explain a bit more what use case you are thinking of, so that I can better answer this question? i.e., what is the few-shot context you are thinking about, what do you want to optimize?

[fivejjs]

After checking the tutorial notebooks, I think one simplified approach could be stacking DSPy's with textgrad. DSPy output further optimized by textgrad. I can create a notebook for that approach but need figure out the textgrad interlas

[ajms]

I am also currently exploring textgrad and I extended the LLMCall and BlackboxLLM classes adding additional variables to the forward methods. These can e.g. contain the fewshot examples. For first tests this works reasonably well.

[mertyg]

Awesome! Thanks to you both for taking the time!! Really appreciate it.

Yeah it should certainly be possible to treat demonstrations as variables and optimize them. Many things to explore here: And certainly using DSPy as an optimizer here, or TextGrad as an optimizer for DSPy programs, or even doing something like coordinate ascent using both optimizers -- are all promising directions!

[dtanow]

I think the easiest might be to dump all examples in the initial prompt and let textgrad "differentiate" over them?

[hamxa678]

I am also currently exploring textgrad and I extended the LLMCall and BlackboxLLM classes adding additional variables to the forward methods. These can e.g. contain the fewshot examples. For first tests this works reasonably well.

Can you explain your point with an example code please?

[ajms]

I made these some time ago, haven't checked if they work with the current version though:

import logging

import jinja2
from textgrad import Variable
from textgrad.autograd.function import BackwardContext
from textgrad.autograd.llm_ops import LLMCall
from textgrad.config import SingletonBackwardEngine
from textgrad.defaults import VARIABLE_OUTPUT_DEFAULT_ROLE
from textgrad.engine import EngineLM, get_engine
from textgrad.model import BlackboxLLM

logger = logging.getLogger(__name__)


class LLMCallEnhanced(LLMCall):

    def forward(
        self,
        input_variable: Variable,
        system_prompt_variables: dict[str, Variable] | None = None,
        response_role_description: str = VARIABLE_OUTPUT_DEFAULT_ROLE,
    ) -> Variable:
        """
        The LLM call. This function will call the LLM with the input and return the response, also register the grad_fn for backpropagation.

        :param input_variable: The input variable (aka prompt) to use for the LLM call.
        :type input_variable: Variable
        :param response_role_description: Role description for the LLM response, defaults to VARIABLE_OUTPUT_DEFAULT_ROLE
        :type response_role_description: str, optional
        :return: response sampled from the LLM
        :rtype: Variable

        :example:
        >>> from textgrad import Variable, get_engine
        >>> from textgrad.autograd.llm_ops import LLMCall
        >>> engine = get_engine("gpt-3.5-turbo")
        >>> llm_call = LLMCall(engine)
        >>> prompt = Variable("What is the capital of France?", role_description="prompt to the LM")
        >>> response = llm_call(prompt, engine=engine)
        # This returns something like Variable(data=The capital of France is Paris., grads=)
        """
        # TODO: Should we allow default roles? It will make things less performant.
        system_prompt_value = self.system_prompt.value if self.system_prompt else None

        # AS: added template rendering for system_prompt
        if system_prompt_variables:
            template = jinja2.Template(system_prompt_value)
            system_prompt_value = template.render(
                **{k: v.value for k, v in system_prompt_variables.items()}
            )

        # Make the LLM Call
        response_text = self.engine(
            input_variable.value, system_prompt=system_prompt_value
        )

        # Create the response variable
        response = Variable(
            value=response_text,
            predecessors=(
                [self.system_prompt, input_variable]
                if self.system_prompt
                else [input_variable]
            ),
            role_description=response_role_description,
        )

        logger.info(
            "LLMCall function forward",
            extra={
                "text": f"System:{system_prompt_value}\nQuery: {input_variable.value}\nResponse: {response_text}"
            },
        )

        # Populate the gradient function, using a container to store the backward function and the context
        response.set_grad_fn(
            BackwardContext(
                backward_fn=self.backward,
                response=response,
                prompt=input_variable.value,
                system_prompt=system_prompt_value,
            )
        )

        return response


class BlackboxLLMEnhanced(BlackboxLLM):
    def __init__(
        self,
        engine: EngineLM | str | None = None,
        system_prompt: Variable | str | None = None,
    ):
        """
        Initialize the LLM module.

        :param engine: The language model engine to use.
        :type engine: EngineLM
        :param system_prompt: The system prompt variable, defaults to None.
        :type system_prompt: Variable, optional
        """
        if (engine is None) and (SingletonBackwardEngine().get_engine() is None):
            raise Exception(
                "No engine provided. Either provide an engine as the argument to this call, or use `textgrad.set_backward_engine(engine)` to set the backward engine."
            )
        elif engine is None:
            engine = SingletonBackwardEngine().get_engine()
        if isinstance(engine, str):
            engine = get_engine(engine)
        self.engine = engine
        if isinstance(system_prompt, str):
            system_prompt = Variable(
                system_prompt,
                requires_grad=False,
                role_description="system prompt for the language model",
            )
        self.system_prompt = system_prompt
        self.llm_call = LLMCallEnhanced(self.engine, self.system_prompt)

    def forward(
        self, x: Variable, system_prompt_variables: dict[str, Variable]
    ) -> Variable:
        """
        Perform an LLM call.

        :param x: The input variable.
        :type x: Variable
        :return: The output variable.
        :rtype: Variable
        """
        return self.llm_call(x, system_prompt_variables)