eval-dev-quality
eval-dev-quality copied to clipboard
Rethink retry logic for LLM Providers
This image shows the uptime of the RWKV v5 World 3B model:
This model is so long down that the retry logic we are currently using does not really work, but waiting for the model to be available again would only stretch out the evaluation runs artificially