prompt2model autopilot: automation of remaining design choices

[neubig]

The prompt2model CLI demo is largely automated, you can put in a prompt and it walks you through the steps to get a model. However, there are still some choices that need to be done manually:

1. Dataset choice: Which dataset of the retrieved datasets to use?
2. Dataset column choice: Which columns in the retrieved dataset to use as input/output
3. Model choice: Which base model of the various base models to use?
4. Parameter choice: Which parameters to use for training, data generation, etc.

It would be good to have an "autopilot" mode where all of these are done automatically. This is of various degrees of difficulty:

• [ ] For dataset and model choice, we can just use the top-1 retrieved dataset/model. This may not be optimal, but it's an easy choice, and as our dataset/model retrievers get better it will hopefully become less of a problem.
• [ ] For parameter choice, as mentioned in https://github.com/neulab/prompt2model/issues/264 , we can choose good defaults. Alternatively, we could try to choose these defaults based on the prompt or constraints.
• [ ] For dataset column choice, this is a bit tricky, as each dataset has columns named in different ways. Maybe we should train a model to choose the best columns and run it over all the datasets to create a "default columns" file?