automate subcommand dispatching (case analysis)

[jnicklas]

I don't think there's any case where several of the boolean flags can be true at the same time is there? With Struct variants being un-feature-gated recently, wouldn't they be a rather nice option?

docopt!(Args deriving Show, "
Usage:
  partners list
  partners set <nick> [--local]
")

would generate something like:

enum Args {
  List,
  Set { arg_nick: String }
}

This would make pattern matching on the parsed arguments very nice:

match args {
  List => list(),
  Set { arg_nick: nick } => set(nick.as_slice())
}

If arguments and options should be namespaced, we could do something like:

struct SetArguments { nick: String }
struct SetOptions { local: bool }
enum Args {
  List,
  Set { arguments: SetArguments, options: SetOptions }
}

But this wouldn't lead to nearly as nice pattern matching.

Just an idea, no idea if this is feasible or not.

[jnicklas]

Just realized that the pattern matching doesn't really work that nicely anyway, since obviously all fields need to be mentioned and there could potentially be a lot of them. Probably better to just add an enum field to the Args struct and be able to match on that.

[BurntSushi]

Indeed that would be very cool and I briefly thought about this. Unfortunately, the entire enterprise is assuming that each Docopt usage pattern can only contain a single command unique to that pattern. Indeed, this is not the case. Commands can be mixed and matched in any way you like. For example, this is a valid Docopt usage string:

Usage:
  prog cmda cmdb
  prog cmdb cmda
  prog cmda ... cmdb

In which case, you get the following parses:

[andrew@Liger docopt_macros] ./scratch cmda cmdb
Args { cmd_cmdb: true, cmd_cmda: 1 }                                                           
[andrew@Liger docopt_macros] ./scratch cmdb cmda
Args { cmd_cmdb: true, cmd_cmda: 1 }                                                           
[andrew@Liger docopt_macros] ./scratch cmda cmda cmda cmda cmdb
Args { cmd_cmdb: true, cmd_cmda: 4 }

It's hard to make commands enums in the general case. There may be a special case lurking somewhere, but I'd be very weary about adding support for it depending on its complexity.

[icorderi]

I think the need to generalized something like is there. It's a very common case to have a binary on which you can call a number of sub-commands.

Take cargo for instance. There is a lot of generic boilerplate code to handle multiple sub-commands that could be abstracted out.

At the end of the day, we should be able to define a USAGE like this:

Rust's package manager

Usage:
    cargo <command> [<args>...]
    cargo [options]

Options:
    -h, --help       Display this message
    -V, --version    Print version info and exit
    --list           List installed commands
    -v, --verbose    Use verbose output

Some common cargo commands are:
    build       Compile the current project
    clean       Remove the target directory
    doc         Build this project's and its dependencies' documentation
    new         Create a new cargo project
    run         Build and execute src/main.rs
    test        Run the tests
    bench       Run the benchmarks
    update      Update dependencies listed in Cargo.lock

See 'cargo help <command>' for more information on a specific command.

And be able to use either a match command {} syntax or have it do a dispatch on the correct execute_command method somehow.

[BurntSushi]

I'm not questioning the need. :-) It's clear to me that some kind of enum would be great. I just don't know how to do it within Docopt. Minimally, my previous comment has to be addressed.

Note though that Cargo's particular case can be done today with an enum precisely because it uses <command> (which is a positional argument in Docopt) as opposed to command (which is an actual command).

I just pushed an example of how it works.

Example use:

[andrew@Liger docopt.rs] ./target/examples/cargo build
Args { arg_command: Build, arg_args: [], flag_list: false, flag_verbose: false }                                                                                                               
[andrew@Liger docopt.rs] ./target/examples/cargo wat
Could not match 'wat' with any of the allowed variants: [Build, Clean, Doc, New, Run, Test, Bench, Update]

[BurntSushi]

There is a lot of generic boilerplate code to handle multiple sub-commands that could be abstracted out.

If you're referring to its use of macros, then I don't think that's related here. The macro is encapsulating case analysis, which won't go away with an enum. There is also some interesting code that tries to guess which command you meant when you make a typo. I think this is still doable with an enum, but you'd have to provide your own Decodable impl (since Docopt's default enum decoder will just fail if no variants match).

[icorderi]

I think that example helps a lot, thanks. How would you get the args for each command in there? Does this work?

#[deriving(Decodable, Show)]
enum Command {
    Build { flag_release: bool, flag_help: bool }
    //...
}

The boilerplate I was referring to is being able to do the following:

docopt::add_command!(BuildArgs, Command::build, BUILD_USAGE);
// or something like 
docopt::add_command_fn!(BuildArgs, Command::build, BUILD_USAGE, ::somewhere::on_build);
// The signature of on_build should be (args: BuildArgs) -> ()

// and then from the main do
fn main() {
    let args: Args = Docopt::new(USAGE)
                            .and_then(|d| d.decode())
                            .unwrap_or_else(|e| e.exit());
    if args.arg_command.is_some(): 
        args.arg_command.dispatch();

    // Obviously in this case, arg_command got augmented 
    // with some trait by the add_command_fn macro.
}

With something like add_command_fn we don't really have to use an Enum to back the options up. This in fact gives us some flexibility to change the options at runtime.

The find_closest() is very a very nice addition that we could reuse with something like:

docopt::enable_suggestions!();

[BurntSushi]

Does this work?

No. That's the part that requires thought (specifically, using struct variants). I suspect it would start with something like... "If each pattern begins with a command name and ..., then ..." The "then" part would likely significantly diverge from what currently happens, which is what causes me pause. It could also result in very uintuitive behavior if the user expects the flags of each sub-command to be local to that sub-command (Docopt chooses the most specific type for each item based on its use in all patterns).

With something like add_command_fn we don't really have to use an Enum to back the options up. This in fact gives us some flexibility to change the options at runtime.

I don't think I fully understand your proposed solution (or how it's an improvement over the status quo). I'm also kind of curious what you think those macros are actually doing. You still need to tease out all of the commands at some point. Whether it's through trait impls, closures or explicit case analysis, you still have to do it.

In my own CLI app, I defined Command as an enum, then I dispatch manually, which in turn calls docopt again. I am not 100% happy with how I've done it, but it does provide a nice clean separation between sub-commands and the "driver" program. (Notably though, the interface of a sub-command is merely implicit!)

Docopt's weakness is that it is a black box (and that box is insidiously complex with subtle semantics). Any solution we find to this problem should, ideally, be implementable outside of Docopt proper. (That doesn't mean it shouldn't be in this crate though! Just an estimate of the abstraction boundary.)

The find_closest() is very a very nice addition that we could reuse with something like:

I wouldn't be opposed to that. I suspect it would slide in nicely to the enum decoding.

[icorderi]

Seems we can agree on what the use case looks like. Questions now is how we want to surface that to the library user...

I think if we can find how we want the user to express it then we can get working on the internals. Even if we come up with a different way to express it later. I think from an architecture point of view, we have two abstractions: a Dispatcher and a Command.

let USAGE = "
Usage:
    example <action> [<args>...]
    example [options]

Some common example actions are:
    jump       How high can you jump?
    run        How fast do you run?
"

<action> can be abstracted as a Dispatcher while jump and run can be abstracted as Command

What-if we could do something like this?

// This is the root
struct Args {
    arg_action: Action,
    flag_help: bool, // do we need to make this flag explicit?
    flag_version: bool,
}

//[#docopt::dispatcher!] // not sure if this does what i think it might do 
enum Action {
    jump(JumpArgs),
    run(RunArgs),
}

[#docopt::dispatcher!(Action)] 

struct JumpArgs { 
    arg_height: uint
}

impl docopt::Command for JumpArgs {
    fn execute(self) -> CliResult<()> {
        // do stuff
    }
}

struct RunArgs { 
    arg_speed: float
}

// We can also throw in this for the lazy people
[#docopt::command!(RunArgs, some_fun, JUMP_USAGE)]

The macros expand to the following:

// [#docopt::dispatcher!(Action)] gets expanded to:
impl docopt::Dispatcher for Action {
    fn dispatch(cmd: Action) ->  CliResult<()> {
        match cmd {
            jump(args) => args.execute(),
            run(args) => args.execute()
        }
    }
    fn usage(cmd: Command) ->  str {
        // same thing
    }
}

// [#docopt::command!(RunArgs, some_fun, RUN_USAGE)] exanding it to this
impl docopt::Command for RunArgs {
    fn execute(self) -> CliResult<()> {
        some_fun(self)
    }

    fn usage() -> str { RUN_USAGE }
}

We can then bake a lot of sugaring on top of those two abstractions. @BurntSushi what do you think?

[BurntSushi]

@icorderi I think I like it. It seems the key contribution is, "if you use this macro, we'll do the command case analysis and dispatch for you" which is really the tedious part.

I think I generally like the direction this is headed toward, but I'm not sure when I'll actually get around to it. Part of it is that once Rust 1.0 hits, syntax extensions will be unavailable on the stable channel, so I'd like to focus my attention on non-macro things. (The docopt! macro was conceived of before the 1.0 release rules. You'll note that it is now down-played in the docs!)

If there's another approach that doesn't involved syntax extensions, then maybe I'll pursue that. But the key is doing the case analysis for you automatically, which I think is going to require some magic.

[icorderi]

@BurntSushi I totally understand the priorities. Without macros is gonna be a tough one.

If there's another approach that doesn't involved syntax extensions, then maybe I'll pursue that. But the key is doing the case analysis for you automatically, which I think is going to require some magic.

The command!() is just sugaring a bit the Command trait implementation, so that can be left out to the user to write the impl. Like you pointed out, the real question is whether or not the Dispatcher can be implemented generically and hidden behind a default implementation. My guess is the answer to that is no.

#[deriving(Dispatcher)] // this would be cool
enum Action {
    jump(JumpArgs),
    run(RunArgs),
}

// somewhere inside docopt
impl Dispatcher  {
    // super hack, I doubt this is remotely possible
    // I don't know if Self works on where clauses... 
    // I would very surprised if Enum even exists, but it would be so cool...
    fn dispatch<T : Self + Enum>(cmd: T) ->  CliResult<()> {
        // what goes in here?
        // for v in cmd.get_variants() ?
        // I haven't read anything on reflection in Rust so I doubt it..
    }
    fn usage<T : Self + Enum>(cmd: T)  ->  str {
        // same deal
    }
}

Btw, is docopt going to have an unstable dist channel?

[BurntSushi]

Btw, is docopt going to have an unstable dist channel?

Hmm. I don't really know how it's going to play out. Cargo depends on Docopt, so to a first approximation, I'll do what's necessary to keep that relationship running smoothly. To a second approximation, I want lots of people to use it.

To my knowledge, external libraries won't be subject to the same restrictions as crates distributed with Rust proper. So I'm not sure if an unstable version is necessary.

On the other hand, moving from experimental to unstable would be nice.

[icorderi]

@BurntSushi and for those following up on this. I've been playing with the idea and I've got some pieces of it working.

• You can find some initial Command and Dispatcher traits here.
• You can check how a module defining a new command looks like here.
• The cmd! macro magic.

The idea is for cli and cli_macros to be generic and not be inside my project obviously.

The next step is to get a procedural macro going to automate this dispatch code into the following:

#[deriving(Decodable, Show)]
pub enum Command {
    Help,
    Write,
    Info,
}

cmd_dispatcher!(Command)
// which will also expand the cmd!(HelpArgs, execute, USAGE)
// instead of having to write them on each module file

Bonus points: Registering the macro as a deriving so we can do #[deriving(CliCommand)]

[keleshev]

I have not really followed this thread, but I saw a word dispatch mentioned, so I figured I'll paste a link here: https://github.com/keleshev/docopt-dispatch

[icorderi]

@keleshev that's python :D Function decorators are slightly more straightforward than messing with the AST with procedural macros in Rust.

[tksfz]

Hi @icorderi I've been emulating your code in a project of mine. Did you ever happen to find a way to make it such that the dispatch match returns e.g. a trait object (a Command instance)? Then you wouldn't have to duplicate the invocations of x.execute(), you could do it generically for all commands. You could also invoke usage() that way etc.

I'm having trouble doing it, but I'm new to Rust and it's leading me down a path of trait object / generics confusion.

[tksfz]

Btw I was able to achieve what I described above using a combination of Box<Command> and UFCS.

[Ericson2314]

https://github.com/docopt/docopt.rs/blob/master/src/parse.rs#L617-L623 that is a good model :). Maybe an enum can be generated for every alternative with disjoint variants?

edit: thinking a bit morel, seeing that a pattern is an grammar of some sort, there should be away to tell if it is ambiguous (disjoint alternatives is part of it). If it isn't, I think a rich type can be generated.

[BlacklightShining]

I would definitely like to see subcommands-as-enums happen. Maybe a Vec of enum variants could be returned in programs where multiple subcommands can be specified at once? If per-subcommand options are required, those could be passed out as associated values.