AttributeError : HARM_CATEGORY_DEROGATORY

[andytan0051]

I installed the framework using pip and somehow received this error while following the documentation.

from trustllm.generation.generation import LLMGeneration

llm_gen = LLMGeneration(
    model_path="'mistralai/Mistral-7B-Instruct-v0.1'", 
    test_type="truthfulness", 
    data_path="dataset",
    online_model=False, 
    use_deepinfra=False,
    use_replicate=False,
    repetition_penalty=1.0,
    num_gpus=1, 
    max_new_tokens=512, 
    debug=False,
    device='cuda:0'
)

llm_gen.generation_results()

I would appreciate some guidance :) Thanks!

[andytan0051]

I believe it is due to the deprecation of PALM models.

[sudarsun]

Yes, currently, only the Gemini family types are supported, which are the following: HARM_CATEGORY_HARASSMENT HARM_CATEGORY_HATE_SPEECH HARM_CATEGORY_SEXUALLY_EXPLICIT HARM_CATEGORY_DANGEROUS_CONTENT HARM_CATEGORY_UNSPECIFIED

[CaptiveCoder]

I want to run privacy evaluations , but am having the same issue as @andytan0051 when I try to import. Any update on a workaround or fix?

[andytan0051]

@CaptiveCoder Try changing the safety setting defined in trustllm_pkg/trustllm/utils/generation_utils.py to the categories mentioned by @sudarsun above. That should solve the problem. Also if possible, install the framework through git clone instead of pip. The pip installed version is outdated and some bugs are left in there.

[HowieHwong]

I want to run privacy evaluations , but am having the same issue as @andytan0051 when I try to import. Any update on a workaround or fix?

Hi,

Please use git clone instead of pip install. The pip install is now out of update.