Guidance on use case

[fozziethebeat]

I want to try using DSPy to optimize some sub-instructions for a model with a very specific prompt format.

Specifically, I want to optimize usage of Prometheus Eval where it has the following fixed format:

###Task Description:
An instruction (might include an Input inside it), a response to evaluate, a reference answer that gets a score of 5, and a score rubric representing a evaluation criteria are given.
1. Write a detailed feedback that assess the quality of the response strictly based on the given score rubric, not evaluating in general.
2. After writing a feedback, write a score that is an integer between 1 and 5. You should refer to the score rubric.
3. The output format should look as follows: \"Feedback: (write a feedback for criteria) [RESULT] (an integer number between 1 and 5)\"
4. Please do not generate any other opening, closing, and explanations.

###The instruction to evaluate:
{orig_instruction}

###Response to evaluate:
{orig_response}

###Reference Answer (Score 5):
{orig_reference_answer}

###Score Rubrics:
[{orig_criteria}]
Score 1: {orig_score1_description}
Score 2: {orig_score2_description}
Score 3: {orig_score3_description}
Score 4: {orig_score4_description}
Score 5: {orig_score5_description}

###Feedback: 

Here, I'd like to try and learn values for {orig_criteria} and {orig_scoreN_description} using DSPy.

I assume this would work with CORPO to just generate better instructions but I'm not quite sure how I'd constrain CORPO to only change those fields in the required format. I'm hoping it can be done using signatures with with fixed formats but i'm not 100% sure how to do that.

If anyone's done something like this, pointing me to an example would be helpful as I try to read through all the existing examples.