camel icon indicating copy to clipboard operation
camel copied to clipboard

feat: Implement structured output by using outlines

Open Appointat opened this issue 7 months ago • 1 comments

This PR introduces structured output capabilities for both open-source and closed-source language models (LLMs) in our system:

  1. Open-source LLMs:

    • Implemented a schema model to integrate open-source, self-hostable LLMs supported by Outlines
    • This enables structured output for open-source models
    • Currently supports platforms: Transformers, LlamaCpp, and vLLM
  2. Closed-source LLMs with function calling:

    • Utilized Pydantic to pass expected output structures
    • This approach enables structured output for closed-source models that support function calling

Key changes:

  • Added SchemaModel class for open-source LLM integration
  • Implemented platform-specific adapters for Transformers, LlamaCpp, and vLLM
  • Created PydanticOutputParser for closed-source LLMs with function calling support

Testing:

  • Added unit tests for SchemaModel and PydanticOutputParser

Next steps:

  • Expand support for additional open-source LLM platforms
  • For the closed llms, we will optimize the issue of multi-functions calling at the same time.
  • Better ci/cd for the self-hosted models

Appointat avatar Jul 17 '24 19:07 Appointat