Structured Output

[Joy-less]

What would you like to be added:

Access to the new structured outputs (JSON response formatting): https://ollama.com/blog/structured-outputs

Why is this needed:

The current JSON formatting doesn't ensure the response follows a certain schema.

[DustinReagan]

I have this working locally, but ~~seems like the repo does not support contributions?~~ I need to fork it first and create a PR.

PR: https://github.com/tryAGI/Ollama/pull/92

[DustinReagan]

to get it working locally, in openapi.yaml replace the current ResponseFormat with:

ResponseFormat:
      oneOf:
        - type: string
          enum: ['json']
          description: "Enable JSON mode by setting the format parameter to 'json'. This will structure the response as valid JSON."
        - type: object
          description: "A JSON Schema object that defines the structure of the response. The model will generate a response that matches this schema."
          additionalProperties: true

Then, in generate.sh comment out these lines:

curl -o openapi.yaml https://raw.githubusercontent.com/davidmigloz/langchain_dart/main/packages/ollama_dart/oas/ollama-curated.yaml
dotnet run --project ../../helpers/FixOpenApiSpec openapi.yaml

and run the script ./generate.sh.

Now you can use an object representation of the schema for the format parameter:

For example, here's the test I wrote for the feature:

public partial class Tests
{
    [TestMethod]
    public async Task StructuredOutputInChat()
    {
#if DEBUG
        await using var container = await Environment.PrepareAsync(EnvironmentType.Local, "llama3.2");
#else
        await using var container = await Environment.PrepareAsync(EnvironmentType.Container, "llama3.2");
#endif

        var chat = container.ApiClient.Chat(
            model: "llama3.2",
            systemMessage: "You are a helpful weather assistant."
        );

        // can also use NewtonSoft.Json.Schema or some other auto schema generation library
        var schemaObject = JsonSerializer.Deserialize<object>(@"{
            ""description"": ""The response to a query about the weather"",
            ""type"": ""object"",
            ""required"": [""Temperature"", ""Location"", ""Unit""],
            ""properties"": {
                ""Temperature"": {
                    ""type"": ""integer""
                },
                ""Location"": {
                    ""type"": ""string""
                },
                ""Unit"": {
                    ""type"": ""string"",
                    ""enum"": [""Celsius"", ""Fahrenheit""]
                }
            }
        }");

        try
        {
            Console.WriteLine($"schemaObject: {schemaObject}");
            chat.ResponseFormat = new ResponseFormat(schemaObject);
            var response = await chat.SendAsync("What is the current temperature in Dubai, UAE in Celsius? (hint: it's 25C in Dubai right now)");
            Console.WriteLine(response);
            var queryResponse = JsonSerializer.Deserialize<QueryResponse>(response.Content);
            queryResponse.Should().NotBeNull();
            queryResponse.Temperature.Should().Be(25);
            queryResponse.Location.Should().Contain("Dubai");
        }
        finally
        {
            Console.WriteLine(chat.PrintMessages());
        }
    }

    [JsonConverter(typeof(JsonStringEnumConverter))]
    public enum TemperatureUnit
    {
        Celsius,
        Fahrenheit
    }

    public class QueryResponse {

        public int Temperature { get; set; }
        public TemperatureUnit Unit { get; set; }
        public string Location { get; set; }
    }
}

You can use NewtonSoft.Json.Schema to generate the schema from the response type automatically, or just manually build it (as I do in the test above).

[DustinReagan]

Unfortunately, it seems that there is an issue with using both tools & structured output together in the ollama api itself: https://github.com/ollama/ollama/issues/8095

[lucyyyyyyy]

Unfortunately it seems like this is currently the expected behaviour from the Ollama API, as per the latest update on that issue.

[fpavlic]

Can someone please provide an example how to use ResponseFormat correctly.

I tried different things in C#, none of them work, some even throw 500 internal server error from ollama.

I have tried also above example from Dustin using curl, and that worked completely fine, but with nuget I'm having issue.

[Joy-less]

Can someone please provide an example how to use ResponseFormat correctly.

I tried different things in C#, none of them work, some even throw 500 internal server error from ollama.

I have tried also above example from Dustin using curl, and that worked completely fine, but with nuget I'm having issue.

You can try using OllamaSharp or Microsoft.Extensions.AI.Ollama, both of which support structured output.