haystack-core-integrations icon indicating copy to clipboard operation
haystack-core-integrations copied to clipboard

Standardize streaming and function calling (FC) compatibility across all generators

Open vblagoje opened this issue 8 months ago • 0 comments

Currently, FC is not supported when streaming is enabled. This limitation affects all generators using streaming capabilities.

Proposed Solutions:

  1. Enable streaming for FC across all generators:

    • Implement function calling to work with streaming in all generators
    • Ensure thorough testing
  2. Disable streaming for FC across all generators:

    • Automatically disable streaming when function calling is required
    • Provide clear documentation explaining why streaming is disabled for function calling, outlining the technical limitations and performance considerations

Action Required:

  • Decide whether to enable streaming for function calling or to disable it and update generators accordingly
  • Update documentation to reflect the changes and provide necessary explanations for users

Documentation Update:

  • Include a section in the documentation explaining the incompatibility between streaming and function calling
  • Detail the reasons for disabling streaming when function calling is enabled focusing on technical constraints

vblagoje avatar Jun 18 '24 07:06 vblagoje