Globs support for defining packages in workspaces

[spydon]

I think this is already planned, but I'm opening an issue so that it can be tracked.

Adding globs support will:

• reduce a lot of user mistakes (like forgetting to add new packages to the workspace)
• make migrations from the current melos version to the future melos version where packages are instead defined in the workspace much easier, since their current definitions could just be moved from melos.yaml to the workspace config file

Design doc discussion: https://docs.google.com/document/d/1UEEAGdWIgVf0X7o8WPQCPmL4LSzaGCfJTM0pURoDLfE/edit?resourcekey=0-c5CMaOoc_pg3ZwJKMAM0og&disco=AAAA9uS5yhA

[jonasfj]

I think add globs in workspace: might be nice, but perhaps it is a follow up enhancement. While it does seem feasible, it definitely will involve some work making sure we can still efficiently validate package_config.json using timestamps.

At-least I don't want it to block launching :rocket:

[spydon]

At-least I don't want it to block launching 🚀

Makes sense, I think we'll wait with putting up a migration from Melos to Melos + workspaces until this is done then. Because it will be quite a painful experience for all projects with more than a few packages.

[sigurdm]

Yeah - I don't think this will make it for 3.6 .

[spydon]

I just started migrating over monorepos now when Melos has support for pub workspaces. Just posting this here to visualize what it looks like today, and how it would look like with globs for Flame (which is not a that big monorepo compared to many other ones out there):

workspace:
  - doc/flame/examples
  - doc/tutorials/klondike/app
  - doc/tutorials/space_shooter/app
  - doc/tutorials/platformer/app
  - packages/flame_behavior_tree
  - packages/flame_behavior_tree/behavior_tree
  - packages/flame_behavior_tree/example
  - packages/flame_test
  - packages/flame_test/example
  - packages/flame_tiled
  - packages/flame_tiled/example
  - packages/flame_oxygen
  - packages/flame_oxygen/example
  - packages/flame_isolate
  - packages/flame_isolate/example
  - packages/flame_texturepacker
  - packages/flame_texturepacker/example
  - packages/flame_lint
  - packages/flame_sprite_fusion
  - packages/flame_sprite_fusion/example
  - packages/flame_devtools
  - packages/flame_fire_atlas
  - packages/flame_fire_atlas/example
  - packages/flame_audio
  - packages/flame_audio/example
  - packages/flame_studio
  - packages/flame_spine
  - packages/flame_spine/example
  - packages/flame_splash_screen
  - packages/flame_splash_screen/example
  - packages/flame_bloc
  - packages/flame_bloc/example
  - packages/flame_kenney_xml
  - packages/flame_kenney_xml/example
  - packages/flame
  - packages/flame/example
  - packages/flame_lottie
  - packages/flame_lottie/example
  - packages/flame_markdown
  - packages/flame_markdown/example
  - packages/flame_console
  - packages/flame_console/example
  - packages/flame_rive
  - packages/flame_rive/example
  - packages/flame_forge2d
  - packages/flame_forge2d/example
  - packages/flame_noise
  - packages/flame_riverpod
  - packages/flame_riverpod/example
  - packages/flame_svg
  - packages/flame_svg/example
  - packages/flame_network_assets
  - packages/flame_network_assets/example
  - packages/flame_3d
  - packages/flame_3d/example
  - packages/flame_jenny
  - packages/flame_jenny/jenny
  - examples/games/trex
  - examples/games/padracing
  - examples/games/rogue_shooter
  - examples

vs. what it would look like with globs:

workspace:
  - doc/**
  - examples/**
  - packages/**

[sigurdm]

Thanks for the example.

Yeah - this is on one hand much more compact - and I think we can make it work.

But I still think having the explicit listing of packages has its benefits too. You can actually see the packages that are expected to be there...

[spydon]

But I still think having the explicit listing of packages has its benefits too. You can actually see the packages that are expected to be there...

In melos we have a command list which shows all the packages that are currently included, maybe we could have something similar here?

I think the pros heavily outweigh the cons of having globs here, if the user wants to have all their packages explicitly listed they are still free to have it like that even after globs are introduced. The main pro we have seen with having globs in Melos is that without it people forget that they have to do the one extra step of manually adding the package to the list, and it then accidentally results in the package being excluded from the ci pipelines, package linking etc, which might not be obvious to see when it happens, until it breaks.

People also tend to have structured monorepos, so if they add a new package somewhere in the file structure where they expect it to be included as a package in the monorepos it would automatically be covered by the glob, and vice versa for packages that shouldn't be added.

[spydon]

I just realized dart pub workspace list already exists! :smiley:

[pcornelissen]

Thanks for the example.

Yeah - this is on one hand much more compact - and I think we can make it work.

But I still think having the explicit listing of packages has its benefits too. You can actually see the packages that are expected to be there...

Well, I wouldn't force everyone to use globs, but forcing everyone who would benefit from globs to litter the config with tons of lines that you need to write a tool to maintain doesn't sound sensible to me.

[damianololo]

I'm with spydon too. In our project we currently have 100+ internal packages. There are many developers working in this project. Maintaining this without using globs will be difficult. It's easier to add one or two packages to exclude than to remember to add all of them to include

[spydon]

Makes sense, I think we'll wait with putting up a migration from Melos to Melos + workspaces until this is done then. Because it will be quite a painful experience for all projects with more than a few packages.

We did release Melos v7 even with this drawback, and it's the most negative feedback we've heard from the migration from Melos v6 to Melos v7, that they now have to explicitly define every package/app in the root pubspec. Other than that it's been quite smooth!

@jonasfj @sigurdm is the globs feature on your roadmap? It now has almost as many thumbs up as the 4 subsequent issues when ordering after thumbs. 😅 I will be giving a talk about Pub Workspaces on FlutterCon in a week, and it would be good to have an answer for the audience regarding this, since I know this question will pop up from people that have previously used Melos v6.

[sigurdm]

@spydon Thanks for pinging on this - it is not on our current roadmap - but given the amount of attention we might want to reconsider.

A few design questions:

• Would it be enough to have dart pub workspace add which supports globs, but adds the explicit elements.
  • probably not - it would not help people that "forget"
• Would it be enough to allow a single final * in the listing:

workspace:
  - pkgs/*

Or do we need/should we have generic glob support (with **)?

• Do we need a way to exclude some libraries from the globs?

• What should happen if the glob resolves to a directory without a pubspec.yaml - is that an error or to be ignored.

...

[spydon]

Thanks for the quick reply!

• Would it be enough to have dart pub workspace add which supports globs, but adds the explicit elements.
  • probably not - it would not help people that "forget"

Yeah, I don't think so, since it would only solve one of the problems.

• Would it be enough to allow a single final * in the listing:

workspace:
  - pkgs/*

Or do we need/should we have generic glob support (with **)?

I would support full globs directly, because then if you add an example it would also be automatically added.

• Do we need a way to exclude some libraries from the globs?

I would say that it's nice to have, but not required. If a user doesn't want it on the workspace they could just not add resolution: workspace I guess?

• What should happen if the glob resolves to a directory without a pubspec.yaml - is that an error or to be ignored.

I think that it should be ignored, people are using all kinds of nested directory structures.