Improving testing for steerable models

[RobinPicard]

We already have tests for steerable models, but they do no cover all possible situations especially for Transformers that supports a wide range of models. We should not only test initialization with various models, but also run all inference tests (maybe also with all backends). Additionally, we should consider running multiple times the same types of inference tests to improve the probability of catching flaky tests because of the stochastic nature of LLM text generation.

[kudos07]

Thanks for raising this. I can help expand the steerable model tests to include multiple model initializations, inference runs across different backends, and repeated runs to detect flaky generation.

Would you prefer this as an extension to the existing test module, or should I create a separate parametrized test suite to keep it isolated?

[RobinPicard]

Hi @kudos07! Thanks for proposing to work on it. Another element I have not mentioned in the issue but that is interesting would be to cache the HF models to prevent rate limit errors (see this commit in pydantic-ai).

I'm not sure what I prefer yet, but I think relying on parametrization would definitely make sense. Feel free to open a draft PR with improvements on a single aspect of the issue so we can discuss about the testing organization without having to spend time doing everything upfront.