prompttools icon indicating copy to clipboard operation
prompttools copied to clipboard

Published 20 hours ago • verified publisher icon hegelai

Generated Robustness Evaluation Sample Scripts

Open RigvedRocks opened this issue 1 year ago • 5 comments

Hey, I am working on issue #69 and I have generated 2 sample scripts - one involving prompt substitution and the other for auto evaluation. I am using Promptbench in both scripts. Can you please guide me as to how to integrate the scripts into your experiments?

RigvedRocks avatar Dec 28 '23 03:12 RigvedRocks

CLA assistant check
All committers have signed the CLA.

CLAassistant avatar Dec 28 '23 03:12 CLAassistant

Thanks for raising this @RigvedRocks! I'll review it now. When you get a chance, could you fill out the CLA?

steventkrawczyk avatar Dec 28 '23 16:12 steventkrawczyk

I'm not very familiar with promptbench but it looks like you want to run attacks as experiments and use your eval function to evaluate the experiment outputs. You can check out experiment examples to see how we feed prompts into LLMs for testing, and how we pass in eval functions too. Let me know if you have any specific questions about it. Thanks again for looking into this issue!

steventkrawczyk avatar Dec 28 '23 16:12 steventkrawczyk

Sure Mr. Krawcyzk. I will have a look at the implementation of the library and will make changes according to that. When time permits, I will fill out the CLA.

RigvedRocks avatar Dec 28 '23 19:12 RigvedRocks

Hey, I have generated a sample script to evaluate the robustness of the models under the utils folder. Please have a look at it and suggest amends if any.

RigvedRocks avatar Jan 02 '24 06:01 RigvedRocks