Proposal: align syntax between `dart pub add`, remote `dart run`, `dart install`

[sigurdm]

(continuation of discussions in https://github.com/dart-lang/sdk/issues/61588)

Recently we added remote functionality to the dart run command (still not in stable).

The currently landed syntax is:

> dart run <host>/<package>[:<executable>]
> dart run https://pub.dev/pana # example
> dart run --help
Run a Dart program from a file, a local package, or a remote package.

Usage: dart [vm-options] run [arguments] [<dart-file>|<local-package>|<remote-executable> [args]]

<dart-file>
  A path to a Dart script (e.g., `bin/main.dart`).

<local-package>
  An executable from a local package dependency, in the format <package>[:<executable>].
  For example, `test:test` runs the `test` executable from the `test` package.
  If the executable is not specified, the package name is used.

<remote-executable>
  An executable from a remote package. This can be from a hosted package server
  (like pub.dev) or a git repository.

  When running a remote executable, all other command-line flags are disabled,
  except for the options for remote executables. `dart run <remote-executable>`
  uses `dart install` under the hood and compiles the app into a standalone
  executable, preventing passing VM options.

  From a hosted package server:
    <hosted-url>/<package>[@<version>][:<executable>]

    Downloads the package from a hosted package server and runs the specified
    executable.
    If a version is provided, the specified version is downloaded.
    If an executable is not specified, the package name is used.
    For example, `https://pub.dev/[email protected]:dcli_complete` runs the
    `dcli_complete` executable from version 1.0.0 of the `dcli` package.

  From a git repository:
    <git-url>[:<executable>]

    Clones the git repository and runs the specified executable from it.
    If an executable is not specified, the package name from the cloned
    repository's pubspec.yaml is used.
    The git url can be any valid git url.

We also have dart install (added in Dart 3.10) which works as follows:

> dart install --help
Install or upgrade a Dart CLI tool for global use.

Install all executables specified in a package's pubspec.yaml executables
section (https://dart.dev/tools/pub/pubspec#executables) on the PATH. If the
executables section doesn't exist, installs all `bin/*.dart` entry points as
executables.

If the same package has been previously installed, it will be overwritten.

You can specify three different values for the <package> argument:
1. A package name. This will install the package from pub.dev. (hosted)
   The [version-constraint] argument can only be passed to 'hosted'.
2. A git url. This will install the package from a git repository. (git)
3. A path on your machine. This will install the package from that path. (path)

Usage: dart install <package> [version-constraint]
-h, --help          Print this usage information.
    --git-path      Path of git package in repository. Only applies when using a git url for <package>.
    --git-ref       Git branch or commit to be retrieved. Only applies when using a git url for <package>.
    --overwrite     Overwrite executables from other packages with the same name.
-u, --hosted-url    A custom pub server URL for the package. Only applies when using a package name for <package>.

Run "dart help" to see global options.

And we have dart pub add

> dart pub add --help
Add dependencies to `pubspec.yaml`.

Invoking `dart pub add foo bar` will add `foo` and `bar` to `pubspec.yaml`
with a default constraint derived from latest compatible version.

Add to dev_dependencies by prefixing with "dev:".

Make dependency overrides by prefixing with "override:".

Add packages with specific constraints or other sources by giving a descriptor
after a colon.

For example (follow the same format including spaces):
  * Add a hosted dependency at newest compatible stable version:
    `dart pub add foo`
  * Add a hosted dev dependency at newest compatible stable version:
    `dart pub add dev:foo`
  * Add a hosted dependency with the given constraint
    `dart pub add foo:^1.2.3`
  * Add multiple dependencies:
    `dart pub add foo dev:bar`
  * Add a dependency override:
    `dart pub add override:foo:1.0.0`
  * Add a path dependency:
    `dart pub add "foo:{path: ../foo}"`
  * Add a hosted dependency:
    `dart pub add "foo:{hosted: https://my-pub.dev}"`
  * Add an sdk dependency:
    `dart pub add "foo:{sdk: flutter}"`
  * Add a git dependency:
    `dart pub add "foo:{git: https://github.com/foo/foo}"`
  * Add a git dependency with a path and ref specified:
    `dart pub add \
      "foo:{git:{url: ../foo.git, ref: <branch>, path: <subdir>}}"`

Usage: dart pub add [options] [<section>:]<package>[:descriptor] [<section>:]<package2>[:descriptor] ...]
-h, --help               Print this usage information.
    --[no-]offline       Use cached packages instead of accessing the network.
-n, --dry-run            Report what dependencies would change but don't change any.
    --[no-]precompile    Build executables in immediate dependencies.
-C, --directory=<dir>    Run this in the directory <dir>.
    --[no-]example       Also update dependencies in `example/` after modifying pubspec.yaml in the root package (if it
                         exists).
                         (defaults to on)

Run "dart help" to see global options.
See https://dart.dev/tools/pub/cmd/pub-add for detailed documentation.

Proposal:

We propose aligning these commands, such that we have a common way of referring to packages and executables.

We propose the following signature:

dart run <package>[:executable][@<descriptor>]
dart install <package>[@<descriptor>]
dart pub add [dev:]<package>[@<descriptor>]

We propose

• using the @ sign to separate package name from descriptor in all these commands
• dart run should have the executable before the descriptor
• dart run without @<descriptor> is a local package reference, while it is a remote package reference for dart pub add and dart install
• For backwards compatibility:
  • dart pub add should be able to separate the descriptor with either colon or @.
  • the current --options to dart install will be maintained, but options hidden from --help

What is a descriptor

A descriptor here is a concept from pubspec.yaml that describes how to retrieve/resolve a version of a package. Essentially, a descriptor is what you write in the pubspec.yaml after writing the package name, like:

dependencies:
  <packageName>: <descriptor>

(documented at https://dart.dev/tools/pub/dependencies#dependency-sources)

In the simplest form a descriptor can be nothing, meaning any version of the package from pub.dev. In the complex cases a descriptor can be a JSON or YAML with multiple keys and sub-objects. Examples of descriptor for retry@<descriptor>

• retry@ (any, preferably latest version of retry from pub.dev)
• retry@^3.0.0 (version >=3.0.0 <4.0.0 of retry from pub.dev)
• 'retry@{"git": {"url": "https://github.com/jonasfj/retry", "ref": "test-branch"}}'
• 'retry@{"git": {"url": "https://github.com/jonasfj/retry", "tag_pattern": "v{{version}}"}, "version": "^3.0.0"}'
• 'retry@{"hosted": "https://some-package-server.com", "version": "^4.0.0"}'
• 'retry@{"path": "/home/jonasfj/packages/retry"}'

While using a descriptor to specify what packages to run/add/install may seem a bit unfriendly to users, it's important to remember that:

• (A) The vast majority of use-cases people will write (or copy from a tutorial):
  • (A1) retry@ (omitting descriptor, because they want the latest version from pub.dev)
  • (A2) retry@^3.0.0 (specifying version constraint only, because they want something with a version range from pub.dev)
• (B) A few users will want to test things from git or path, or use a custom pub server, for those users using a descriptor will be moderately ugly and complicated. But the syntax is consistent with pubspec.yaml.

Example dart commands

• dart run build_runner build, run bin/build_runner.dart from build_runner (which is a dependency of my current project)
• dart run pana@, run the latest version of pana from pub.dev against my current project.
• dart pub add retry@^3.0.0, add dependency on retry with version range ^3.0.0
• dart install dartdoc, install latest version dartdoc.

Discussion

pros:

• Consistency across commands
• the simple cases are easy (empty descriptor or a version number describes a package from pub.dev)
• you can express anything pub can express (as pub will be the one parsing the descriptors) For example you get compatibility with git tag_patterns for free.
• the @ sign would be a clear visible marker that you are not acting on a local package for dart run.
• the dart run <repository>/<package> syntax looks like a url, but pub.dev would not actually serve the package page at that url.
• We generally don't want people to write the default value for PUB_HOSTED_URL, this allows people to overwrite it without having to update all the packages. Teaching people to write it in dart run https://pub.dev/... is undesirable.
• From an engineering point of view adopting descriptors in CLIs as a (black box) reusable concept, would simplify building abstractions on top of pub. Not every corner case of pub descriptors needs to be manually supported in dart tooling that uses pub somewhere inside and allows users to influence that.
• From a documentation point of view a reusable concept would simply always point to https://dart.dev/tools/pub/dependencies#dependency-sources

cons:

• Typing out a correctly quoted complex descriptor is somewhat hairy. For example: dart run "foo@{hosted_url: onepub.dev, version:^1.0.0<1.3.0}:foo_bin" some arguments to the dart file.
• It can be a bit subtle that @ suffix in dart run foo@ means run foo from pub.dev
• Breaking changes for dart install (shipped in Dart 3.10), or at least supporting both syntaxes for a while.
• ~~Breaking changes for dart run <remote> assuming we don't get this landed before the branch cut of 3.11.~~ Hiding dart run <remote> behind a flag before the Dart 3.11 release.

cc @mit-mit @dcharkes @jonasfj @parlough @bsutton @jakemac53 @bkonyi