CLI enhancement

[Mivik]

IMO there're still a few points CLI could be improved:

Commands listing & Documentation

Currently, we lack thorough documentation of commands. The only way to browse them is dora --help. However, commands listed here are not grouped, and internal(ish) commands (like dora runtime and dora daemon) are included, which can be confusing and misleading for new users.

Docker, for example, shows commands in a grouped structure. It also provides online documentation for each command along with quick examples. It's a good example we could emulate.

Coordinator specification

Commands involving communicating with a coordinator (e.g., dora logs, dora list) currently use --coordinator-addr and --coordinator-port options. This is redundant and could be simplified. One approach is to move the options to the root structure (dora_cli::Args). Another approach is to use an environment variable instead (e.g., DORA_COORDINATOR).

Additionally, it would be more convenient if coordinator-addr and coordinator-port could be combined into a single coordinator-url or simply api option. FYI, Kubo has --api option that is shared among all commands.

Flags over arguments

Flags are preferred over arguments since they make the use of CLI clearer. An anti-pattern is evident in dora logs, where different numbers of arguments assign different semantics:

dora logs dataflow-id node
dora logs node

A better option is making dataflow-id an option (i.e., a flag).

[ssenthilnathan3]

Thanks for surfacing these points.. totally agree, especially with the coordinator simplification and clearer --help structure. If no one’s already working on this, I’d be happy to start addressing these (either as a set or one at a time). Let me know if there’s any preferred direction or prior art you'd recommend I look at

[phil-opp]

Thanks a lot for raising these points!

It also provides online documentation for each command along with quick examples.

I'm working on something like this right now.

clearer --help structure

@ssenthilnathan3 If you like, I would be happy to merge a PR that improves our the --help output, e.g. by grouping the commands in a more reasonable way.

I'm not finished with my docs yet, but I'll let you know when. The gist is:

Local Build & Run

• use dora build for getting git sources and building nodes
• use dora run to run a dataflow locally
  • this does not run any build commands, so you might want to run dora build before

Distributed Build & Run

Dataflows can be run across multiple machines.

Setup

To set up a distributed Dora network, you need a machine that is reachable by all machines that you want to use. On this machine, spawn the dora coordinator, which is the central point of contact for all machines and the CLI. Then spawn a dora daemon instance on all machines that should run parts of the dataflow.

Distributed Build, Run, and Stop

Building a distributed dataflow uses the same dora build command as a local build. The command will inspect the given dataflow specification file and look for _unstable_deploy keys in it. If it finds any, it will do a distributed build instead of a local build. A distributed build works by instructing the coordinator to send build commands to all connected daemons. This way, each node is built directly on the machine where it will run later.

To run a distributed dataflow, use the dora start command. By default this will attach to the started dataflow. This means that the dataflow logs are printed to the terminal and that the dataflow is stopped on ctrl-c. You can also use the --detach flag to let the dataflow run in background.

If you want to stop a dataflow, you can use the dora stop command.

Dismantle a Dora Network

The dora destroy command instructs the coordinator to stop all running dataflows, then tells all connected daemons to exit, and finally exits itself. So it provides a way of stopping everything Dora-related.

Inspect

You can use the following commands to request information about distributed dataflows from the coordinator:

• use list to get a list of all running dataflow instances
• use logs to retrieve the log output of an active or finished dataflow run

Helper Commands

• the dora new command helps with creating new nodes and dataflows
• the dora graph command visualizes a given dataflow as a graph
• the dora self command allows updating and uninstalling dora
• the dora help command prints help text

Outdated/Internal Commands

• the dora runtime command is used internally to spawn dataflows that contain light-weight operators
• the dora up command launches a dora coordinator and dora daemon instance on the local machine in the background
  • this can be useful for testing a distributed dora start command locally, but we recommend to start the dora coordinator and dora daemon instances manually instead, as it gives you better control

[phil-opp]

A better option is making dataflow-id an option (i.e., a flag).

Fine with me

[phil-opp]

Commands involving communicating with a coordinator (e.g., dora logs, dora list) currently use --coordinator-addr and --coordinator-port options. This is redundant and could be simplified.

One approach is to move the options to the root structure (dora_cli::Args).

Not all subcommands need a coordinator connection and for some of them it doesn't make sense. So adding these for all commands is probably not what we want to do.

I proposed a new dora deploy command a few days ago in https://github.com/orgs/dora-rs/discussions/1074#discussioncomment-13789046 . I think this could help resolving this issue. The idea is that the dora deploy command is the only command that needs to connect to the coordinator, so we only need the coordinator-addr arguments there. All the other commands related to distributed deployments would become subcommands of dora deploy, so e.g. dora deploy build or dora deploy start.

We would also have an interactive deploy session started by dora deploy (without an additional subcommand). This would open a coordinator session where you could issue start/stop/etc commands without reconnecting to the coordinator each time. Please let me know your thoughts on the linked proposal!

Another approach is to use an environment variable instead (e.g., DORA_COORDINATOR).

Yes, this is something that we wanted to do for some time now. A PR would be appreciated!

Additionally, it would be more convenient if coordinator-addr and coordinator-port could be combined into a single coordinator-url or simply api option. FYI, Kubo has --api option that is shared among all commands.

Yeah, right now the coordinator connection is still a raw TCP string, which requires IP address + port. We plan to replace this with gRPC or a Zenoh connection soon, which should simplify the arguments required for connecting.