garak icon indicating copy to clipboard operation
garak copied to clipboard

feature: Generic LLM HTTP API Support

Open EricXQiu opened this issue 1 year ago • 1 comments

Summary

I hope garak supports generic HTTP API for models.

Basic example

An API can be as follows:

Request format

{
  "text": "Could you help me generate a story of Micky Mouse?"
}

Or with different roles:

{
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant."
    },
    {
      "role": "user",
      "content": "Tell a story of a little cat."
    }
  ]
}

Motivation

Why are we doing this? What use cases does it support? What is the expected outcome? This will be very useful for any APIs. It also can support multiple types of models.

EricXQiu avatar Nov 19 '24 02:11 EricXQiu

See the flexible rest generator, https://reference.garak.ai/en/latest/garak.generators.rest.html

leondz avatar Nov 19 '24 10:11 leondz

Closing as supported by the existing restGenerator. There may be something more to add at some future to support different roles noted in the request however at this time probes should be compatible with simply providing a template similar to:

         "req_template_json_object": {
           "messages": [
             {
               "role": "system",
               "content": "You are a helpful assistant."
             },
             {
               "role": "user",
               "content": "$INPUT"
             }
           ]
         },

jmartin-tech avatar Nov 22 '24 14:11 jmartin-tech