No Improvement after using TextGrad for Prompt Optimization

[nikhilk7153]

Hello,

I have currently tweaked the prompt optimization tutorial so that I can see if I can improve it's ability to improve on medical multiple -choice datasets. However, the results are getting progressively worse as I do more iterations. I have set the gradient_memory = 3 so that it can see the previous instances but that is not doing much. Does anyone have any suggestions on what else I can do to improve the score? My evaluation function is just simply checking if there is a match between the correct answer choice and the LLM inference answer choice.

I can try changing the LLM optimizer system prompt but I am not sure how much of a difference that makes. Also, I can share my repo if that helps.

Thanks!

Nikhil

[belmarca]

Can you double check that your prompt gets gradients? In larger examples, mine don't and I get the same behavior.

[vincentWuK]

try to use textgrad with deepeval as loss func