haystack-core-integrations icon indicating copy to clipboard operation
haystack-core-integrations copied to clipboard

fix: normalising ChatGenerators output

Open davidsbatista opened this issue 1 year ago • 3 comments

Related Issues

  • fixes #7687

Proposed Changes:

  • Something change in AmazonBedrock and the PR I merged had no effect

How did you test it?

unit tests, integration tests, manual verification, instructions for manual tests

A few things I noticed when testing the ChatGenarators

Test coverage

  • generators/amazon_bedrock/chat/adapters.pyis at 59%
  • generators/amazon_bedrock/chat/chat_generator.py is at 51%

Some integrations tests are still being skipped, see the CI logs for details

cmd [1] | coverage run -m pytest -m integration --reruns 3 --reruns-delay 30 -x
============================= test session starts ==============================
platform linux -- Python 3.9.19, pytest-8.3.2, pluggy-1.5.0
rootdir: /home/runner/work/haystack-core-integrations/haystack-core-integrations/integrations/amazon_bedrock
configfile: pyproject.toml
plugins: rerunfailures-14.0, anyio-4.4.0
collected 152 items / 140 deselected / 12 selected
tests/test_chat_generator.py::TestMistralAdapter::test_default_inference_params[mistral.mistral-7b-instruct-v0:2] SKIPPED [  8%]
tests/test_chat_generator.py::TestMistralAdapter::test_default_inference_params[mistral.mixtral-8x7b-instruct-v0:1] SKIPPED [ 16%]
tests/test_chat_generator.py::TestMistralAdapter::test_default_inference_params[mistral.mistral-large-2402-v1:0] SKIPPED [ 25%]

Several warnings about parameters being ignored

WARNING  haystack_integrations.components.generators.amazon_bedrock.chat.adapters:adapters.py:76 Parameter 'top_k' is not allowed and will be ignored.
WARNING  haystack_integrations.components.generators.amazon_bedrock.chat.adapters:adapters.py:76 Parameter 'stop_sequences' is not allowed and will be ignored.
WARNING  haystack_integrations.components.generators.amazon_bedrock.chat.adapters:adapters.py:76 Parameter 'stream' is not allowed and will be ignored.

Checklist

davidsbatista avatar Aug 12 '24 16:08 davidsbatista

@vblagoje @anakin87 regarding the issues I raised in the PR comment - I will open an issue for it, you agree?

davidsbatista avatar Aug 13 '24 14:08 davidsbatista

On test coverage, etc.: see #801

In short, we split unit and integration tests to make integration tests run only if authenticated (not from forks).

  • This causes the coverage numbers to be strange and probably incorrect.
  • Selected tests are different depending on the unit/integration workflow step.

anakin87 avatar Aug 13 '24 14:08 anakin87

ok I see the rational do to that, but since I couldn't successfully authenticate locally to test everything, I'm not sure something will fail when the full tests run on main

davidsbatista avatar Aug 13 '24 14:08 davidsbatista

@anakin87 fixed

davidsbatista avatar Aug 19 '24 08:08 davidsbatista

This test is wrong https://github.com/deepset-ai/haystack-core-integrations/blob/55c65af5ac338f0678537125d6fd0e92dfa689b6/integrations/amazon_bedrock/tests/test_chat_generator.py#L231

meta should be a dict: https://github.com/deepset-ai/haystack/blob/35b1215b00b88b2654ea50de2453e5f852e72328/haystack/dataclasses/chat_message.py#L33

I think you can fix the test and add back the previous suggestion.

anakin87 avatar Aug 19 '24 08:08 anakin87