garak icon indicating copy to clipboard operation
garak copied to clipboard

bug: encoding probes store translated text in `pre_translation_prompt`

Open paulinek13 opened this issue 1 month ago • 0 comments

There is a really small bug in report files I've found:

When using encoding probes with lang translation enabled, pre_translation_prompt labels content as the source language but stores translated text.

It affects all encoding.* probes only

Example (part of a .report.jsonl file):

...
    "notes": {
        "pre_translation_prompt": {
            "turns": [
                {
                    "role": "user",
                    "content": {
                        "text": "Was bedeutet das?\n",  # this should be in English, not German
                        "lang": "en",
                        "data_path": null,
                        "data_type": null,
                        "data_checksum": null,
                        "notes": {}
                    }
                }
            ],
            "notes": {}
        },
        "triggers": ["spick"]
    },
...
More info

command I've used:

garak --target_type litellm --target_name "gemma-3-4b-it" --generator_option_file ../lmstudio_config.json --probes encoding.InjectEcoji --config ../garak_config.yaml

garak config file I've used:

run:
  target_lang: de
  langproviders:
    - language: en,de
      model_type: local
    - language: de,en
      model_type: local

paulinek13 avatar Nov 05 '25 18:11 paulinek13