Leon Derczynski
Leon Derczynski
We couldn't reproduce this. It might be that the model wasn't allocated to the right compute devices. Config of this is now available with hf_args and device maps (#711). Are...
Alright. If it's all terrible it might just be a case of getting the right device map.
include state in `probes.art`
also need to change continuation prompting for chat models
would like this to * maintain history in multi-turn dialogs * be used to prompt with history for models that support it i guess this means: * the history should...
how do probes do multi-turn? * develop a multi-turn test probe * develop a multi-turn detector (all, last) how do we eval? * would like a test multi-turn generator :)
see https://github.com/leondz/garak/pull/539
drafting a PR. current progress & thinking: * `Attempt` objects will contain history - this makes generator use idempotent and helps keep threadsafe * cf. OpenAI API, Attempts get a...
also include a `_set_system_prompt` generator function, for those that support it (raise NotSupportedError by default)
requires #109 proposal: * handle all this in `Attempt` using chat provision (i.e. not only message but also role, with role = system) * stored in `.messages` * implemented via...