Improved command-line API

[lebrice]

Need to design and implement a better command-line API for with commands, etc.

Hey @oleksost , @optimass , @prlz77, @pclucas14, please let me know what you think:

Users should be able to easily do these from the command-line

• Evaluate a single Method on a single Setting
• Evaluate a single method on a "benchmark" (pre-configured Setting or multiple settings?)
• Evaluate a single Method on all its applicable Settings
• Evaluate a single Method on a single Setting, but multiple datasets?
• Evaluate all applicable Methods for a given Setting

Here are some ideas:

• sequoia list: lists out all the available settings / methods
• sequoia help <some_method_or_setting>: prints out available command-line arguments / class docstring / some useful documentation about a Method or Setting
• sequoia --benchmark incremental_cartpole --method baseline ?
• sequoia --setting task_incremental --method baseline ? or sequoia task_incremental_rl baseline?
• sequoia --setting task_incremental --method ALL ?
• sequoia --method baseline --setting ALL ?

Please chip in if you have any other ideas, the goal is for this to be simple, intuitive, and flexible.

[TLESORT]

I think using the flag --help might be the more intuitive solution. I don't know how easy it is achievable but it could work like this:

sequoia --help displays all the arguments for sequoia sequoia --one_argument --help display information about the argument use sequoia --one_argument one_value --help displays information about a value that the argument might take

Ex: with the algorithms argument sequoia --algorithms --help display information about the available training algorithms

sequoia --algorithms ewc --help display information about ewc algorithm

[TLESORT]

I don't know what is possible to do with the simple_parsing lib but I think that argparse library might do all of this

[optimass]

+1 on all the examples that the user should be able to launch from the command-line.

+1 for Tim's reframing. Seems pretty intuitive.

Not sure what else to add, sorry. Maybe I can help if you have other, more precise questions.

[lebrice]

Hey @TLESORT just FYI, simple-parsing is just plain old argparse, it doesn't change anything, apart from one method on the ArgumentParser, for adding groups of arguments.