Martin Treml

Not sure if directly related but when I try to run `make test` localy on our main branch the following test fails: `cmd/eval-dev-quality/cmd TestEvaluateExecute/Model_filter/Ollama/Pulled_Model` Local version of ollama is `0.1.34`...

Running `make install-all` did solve the problem for me locally. Will take a look at this Bug.

@bauersimon @zimmski updated the description on "check if model is loaded"

We are installing the `symflower` binary in https://github.com/symflower/eval-dev-quality/blob/main/cmd/eval-dev-quality/cmd/evaluate.go#L185 when we are running an evaluation and for that we are using a helper for the path https://github.com/symflower/eval-dev-quality/blob/main/tools/tools.go#L13. But inside the https://github.com/symflower/eval-dev-quality/blob/main/model/symflower/symflower.go#L34...

Ahhh okay, so only the `tests` are failing because of the missing `symflower` binary. Alright, running the evaluation directly does work as expected but running the tests with `make test`...

- Would throw an error when a name clashes, already existing provider should take precedence (is that even possible? we prefix the "custom" part so there shouldn't be any clashes...

> (even though the listing and validating models is pointless then because the listed ones are the ones specified). Only for the custom provider which I would say is by...

The reporting command should already handle globing

Provider Routing for Openrouter: https://openrouter.ai/docs/provider-routing

This is a derivation of the OpenAI-API which means we need to gate the feature for OpenRouter by either forking the openAI api library we use or switch the API...