Homogenize Generator meta output

[nachollorca]

Is your feature request related to a problem? Please describe. The dictionaries returned by generators have matching keys to an extent, but some are using different notations for the same concept, mainly within the metadata / usage keys. For example:

• OpenAI generators use prompt_tokens/completion tokens, while Cohere/Anthropic/AmazonBedrock use input_tokens/output_tokens.
• Some generators use finish_reason, while others use stop_reason

Describe the solution you'd like I would like generators to output the same scheme and keys to the extent that it is possible, i.e. for matching concepts. In particular, I would prefer to use input_tokens/output_tokens and stop_reason.

Describe alternatives you've considered So far, in apps where we swap generators depending on the use case, the alternative is:

meta = pipe_result["answer_generator"]["replies"][0].meta
usage = meta["usage"]

output.input_tokens = usage["prompt_tokens"] if "prompt_tokens" in usage else usage["input_tokens"]
output.output_tokens = usage["completion_tokens"] if "completion_tokens" in usage else usage["output_tokens"]
output.finish_reason = meta["finish_reason"] if "finish_reason" in meta else meta["stop_reason"]`

[vblagoje]

Taking into account tendency for other LLM providers/libraries to converge on OpenAI API - let's got with OpenAI naming scheme.

[nachollorca]

Alright, I'll go for OpenAI's convention :)

[CarlosFerLo]

Been having a look at all generators and all the keywords seem to keep the OpenAI standard.

[mrm1001]

Hi @nachollorca I am the product manager of Haystack and I like to reach out to users to better understand their use cases and see how we can offer better support. Would you have some 15 min in the next few weeks for a quick chat? https://calendly.com/maria-mestre-ugu/haystack-catch-up

[nachollorca]

@CarlosFerLo

Been having a look at all generators and all the keywords seem to keep the OpenAI standard.

At least Cohere, Anthropic and AmazonBedrock use 'meta': {'model': 'claude-2.1', 'index': 0, 'finish_reason': 'end_turn', 'usage': {'input_tokens': 18, 'output_tokens': 58} instead of prompt_tokens, completion_tokens

[CarlosFerLo]

@nachollorca Hey these Generators are in haystack-extensions I believe, maybe we should move this issue there.

[vblagoje]

Hey @nachollorca @CarlosFerLo - most of the generators are in https://github.com/deepset-ai/haystack-core-integrations/ GitHub project

[davidsbatista]

I will normalise all the generators that return a meta field to have the following keys:

'meta': [
{
  'model': 'claude-3-sonnet-20240229',
  'index': 0, 
  'finish_reason': 'end_turn', 
  'usage': {'prompt_tokens': 16, 'completion_tokens': 55}
}]}

[davidsbatista]

@nachollorca I'm going through every generator and seeing what comes in the meta field,

I don't see what needs to be changed on Cohere:

client = CohereGenerator()
esponse = client.run("Briefly explain what NLP is in one sentence.")
print(response)

{'replies': ['Natural Language Processing (NLP) is the study of natural language and how it can be used and understood by computers.'], 
'meta': [
    {'model': 'command-r', 
     'usage': 33, 
     'index': 0, 
    'finish_reason': 'COMPLETE', 
    'documents': None, 'citations': None}]
}

meta contains only finish_reason - I can't see any reference to tokens either given as input or generated.

What do you what to change?

[davidsbatista]

@nachollorca when you use the word generator are you referring to Generator, ChatGenerator or both?

[davidsbatista]

@nachollorca I will assume the change you want is related to ChatGenertor - I run each one and this is the meta I got from each. See below the output.

I will first add DeprecationWarning to the following:

• AmazonBedrockChatGenerator
• AnthropicChatGenerator

since in the next release we will change the keys in the meta to be in line with OpenAI

AmazonBedrockChatGenerator

meta={
  'id': 'msg_bdrk_01SsZEsJveWVZVy8Uaaih8LS', 
   'model': 'claude-3-sonnet-20240229', 
   'stop_reason': 'end_turn', 
   'stop_sequence': None, 
   'usage': {'input_tokens': 27, 'output_tokens': 147}})]}

AnthropicChatGenerator

meta={
   'model': 'claude-3-5-sonnet-20240620', 
   'index': 0, 
   'finish_reason': 'end_turn', 
   'usage': {'input_tokens': 16, 'output_tokens': 63}})]}

CohereChatGenerator

meta={
    'model': 'command-r', 
    'usage': 81, 
    'index': 0, 
    'finish_reason': 'COMPLETE', 
    'documents': None, 
    'citations': None})]}

HuggingFaceAPIChatGenerator

meta={
    'model': 'HuggingFaceH4/zephyr-7b-beta', 
    'finish_reason': 'eos_token', 
    'index': 0})]}

MistralChatGenerator

meta={
    'model': 'mistral-tiny', 
    'index': 0, 
    'finish_reason': 
    'stop', 'usage': {}})]}

OpenAIChatGenerator

meta={
    'model': 'gpt-3.5-turbo-0125', 
    'index': 0, 
    'finish_reason': 'stop', 
    'usage': {'completion_tokens': 53, 'prompt_tokens': 16, 'total_tokens': 69}})]}

ToDo:

• VertexAIGeminiChatGenerator
• GoogleAIGeminiChatGenerator