prompt2model
prompt2model copied to clipboard
Tuning of model and dataset retrievers
Currently our model and dataset retrievers are not perfect, and it would be good to have a way to make them better.
One way we can do so is by explicitly training the model/dataset retrievers to:
- Retrieve multiple datasets (models) and run the prompt2model pipeline with all of them
- Take the resulting accuracy scores, and train the retriever so that the retriever gives higher scores to datasets (models) that give higher accuracy scores for the full pipeline
This would result in a training objective that explicitly rewards retrieving of datasets (models) that give high accuracy.
This would also be helpful for #285 , as it would reduce the need for human intervention when selecting models.