Enable more automated workflows by allowing publishing from the `workflow_dispatch` trigger

[spydon]

Note: Pub.dev only allows automated publishing from GitHub Actions when the workflow is triggered by pushing a git tag to GitHub. Pub.dev rejects publishing from GitHub Actions triggered without a tag. This ensures that new versions cannot be published by events that should never trigger publishing.

I understand the reasoning here, but this also makes it impossible to do fully automated publishing since a tag that is created by another workflow isn't allowed to trigger other workflows. So with this restriction from pub.dev in place the publishing will always need manual intervention from the command-line/IDE, since a user has to push a tag manually to trigger the workflow.

What we would like is to trigger the workflow that publishes the new versions from another workflow by using the workflow_dispatch trigger (currently only push is allowed).

The simplest flow for our use-case would be this:

1. The users goes to their action tab and presses "Run release action"
2. The melos-action versions and creates changelogs for all packages, and then published a PR with the changes (it also checks if it gets any publish --dry-run warnings in this step)
3. The PR is then reviewed
4. When the PR is merged it triggers the melos publish job which will tag and publish all of newly versioned the packages.

For context this will for example be used by the melos-action that should semi-automate (PRs are still used) the whole publishing flow in monorepos, or any repositories using melos.

Design doc: https://flutter.dev/go/pub-automated-publishing

[spydon]

One solution could be for pub to also allow workflow_dispatch events, they can also be connected to tags. I thought that might already work, but sadly it didn't:

The calling GitHub Action is not allowed to publish, because: publishing is only allowed from 'push' events, this token originates from a 'workflow_dispatch' event.

[spydon]

What do you think @isoos @sigurdm?

[isoos]

@spydon: I am not that familiar with the workflow internals. Is there a document that describes what the users are doing and what the system is doing in such case?

[spydon]

@isoos So workflow_dispatch is the event that is used when a workflow is triggered manually: https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_dispatch

It can also be connected to a tag, so the rest of the server-side checks on your side should still be the same as they are for push.

[isoos]

I'm more interested in what kind of problem it solves. I remember @jonasfj doing interesting things with workflows+tagging+approvals, and I don't know how this related.

[spydon]

Ah sorry, maybe I didn't clarify that properly in the initial post. So with only allowing publishing on push one can never fully automate publishing, or even trigger a workflow manually that handles publishing from GitHub, one always have to manually push tags from the local machine.

What I'm building in the melos-action is so that people can set up the following flow for their monorepos:

1. Automatically version their packages and create changelogs.
2. This creates a PR with the outcome for the maintainers to review.
3. When this release PR is merged, another job creates git tags for all the new versions.
4. After the tags are created it should then publish to pub, but since GitHub doesn't allow workflows to be triggered by the action of other workflows (to avoid infinite loops) this part is not possible, even with an explicit trigger through workflow_dispatch, since pub.dev blocks that server-side.

So it really is just an unfortunate combination of the rule from GitHub and the one from pub.dev that blocks this flow, but by allowing workflow_dispatch events and not only push events to trigger publishing it would be solved.

[isoos]

When this release PR is merged, another job creates git tags for all the new versions.

I've seen people use git tagging and git push --tag to trigger automatic publishing. Not sure: what is missing in it for this to work with your workflow?

[spydon]

I've seen people use git tagging and git push --tag to trigger automatic publishing. Not sure: what is missing in it for this to work with your workflow?

If you do that from a GitHub workflow it can't trigger another action, so you can only do that locally. https://docs.github.com/en/actions/using-workflows/triggering-a-workflow#triggering-a-workflow-from-a-workflow

[spydon]

@isoos to clarify in case it wasn't clear, when you run git push --tags from a workflow, it will create the tags, but GitHub refuses to trigger another workflow that is usually triggered by newly created tags when those tags (or any other trigger, except workflow_dispatch) is related to another workflow run.

[spydon]

@isoos I found a TODO from you about possibly allowing for more event types :smile: https://github.com/dart-lang/pub-dev/blob/master/app/lib/package/backend.dart#L1250-L1251

I can implement the PR so that you can see what it would look like with workflow_dispatch too, it should be a really minimal change.

[isoos]

@spydon: As the TODO says we shall consider it :). I'd wait the input from @jonasfj but he is out right now, it may take a while before we can give you any solid decision on this one.

[spydon]

@spydon: As the TODO says we shall consider it :). I'd wait the input from @jonasfj but he is out right now, it may take a while before we can give you any solid decision on this one.

Makes sense, I'll not put up a PR then, or would it help your decision to have a PR to look at? :)

[jonasfj]

As far as I'm aware there is no work-around for this (without having to use PAT tokens)?

You can use a GCP service account instead. Ideally, not be exporting the service-account keys, but by running on Google Cloud Build or by using Workload Identity Federation to allow your Github JWTs to be used to obtain permission to impersonate the GCP service account.

This is obviously, quite complicated -- just saying it's possible :see_no_evil:

---

On topic we do have concerns, there are reasons we did it the way we did. Some of these concerns include:

• (1) Publishing a new version should be a deliberate action.
  • We do NOT want users publishing a new version of their package for each commit. (It's not hard to imagine that someone would think this is a cool thing to do).
  • pub.dev will limit packages to ~1000 package versions or so, if/when someone hits that we'll have to make exceptions on a case-by-case basis -- which we'll probably have to do.
  • Having many versions will make the API used by pub get slow, we want a simple and fast API.
  • Frequently publishing new versions might cause upgrade fatigue or unnecessary churn for package consumers -- I'm sure there are lots of other good reasons not to have a huge amount of versions.
  • Forcing automated publishing to happen through a tag, makes the initiation somewhat manual. This might be a good thing, and might keep people from publishing on every commit.
• (2) Reduce complexity reduces the risk of misconfigured Github Actions.
  • The fact that publishing requires permission to publish a tag makes it hard to misconfigure your Github Actions such that a pull-request can publish to pub.dev.
  • The fact that Github Actions can be triggered in so many ways, also means that if we allow arbitrary triggers we increase the risk that someone configures it poorly.
• (3) Forcing git revisions to be tagged is really awesome!

I think (3) is really nice. I don't know if (1) could be solved by other means -- maybe, but it's definitely a concern that if we make publishing too easy, someone will publish on every commit.

I think (2) is a really be concern though.

---

I'd be very curious is hear what kind of workflow you envision?

What are the kind of tasks people would use melos for? What are the steps in each scenario?

[spydon]

You can use a GCP service account instead

Isn't this kind of strange, that you allow this to be done through one service, but not another? It wouldn't work here either obviously since it's a GitHub action that we're building for the users, if we don't make something reeeeally hacky. :sweat_smile:

The fact that Github Actions can be triggered in so many ways, also means that if we allow arbitrary triggers we increase the risk that someone configures it poorly.

That is the good thing with workflow_dispatch, the user really has to either actively press "Run action" in the UI, or explicitly write that it should be triggered in their action, it is never triggered by anything else happening in the repository.

(3) Forcing git revisions to be tagged is really awesome!

Melos tags all packages when it creates versions, the problem is that when these tags are created within a pipeline another action is not allowed to be triggered by those tags (https://docs.github.com/en/actions/using-workflows/triggering-a-workflow#triggering-a-workflow-from-a-workflow).

I'd be very curious is hear what kind of workflow you envision?

The simplest flow would be this:

1. The users goes to their action tab and presses "Run release action"
2. The melos-action versions and creates changelogs for all packages, and then published a PR with the changes (it also checks if it gets any publish --dry-run warnings in this step)
3. The PR is then reviewed
4. When the PR is merged it triggers the melos publish job which will tag and publish all of newly versioned the packages.

Number 1 here can be changed to for example writing "Trigger release" or something like that in a previous PR description, but a user will still have the manual step of reviewing the PR with the release in it, which I guess could be compared with the manual intervention of creating a tag today.

[spydon]

What do you think @jonasfj, does it make sense? :)

[clragon]

I have now run into this problem.

This is essentially preventing me from making a single trigger workflow that

• increases my pubspec version
• stamps my changelog
• commits these changes
• creates a github release
• publishes to pub.dev

Because pub.dev only allows push: tags trigger workflows to publish, I use the setup you have provided that publishes on tag push. I have a trigger workflow that does the things mentioned above except publishing. A github workflow that creates a release (read: pushes a tag) with the GITHUB_TOKEN cannot trigger another workflow. So then finally, I have to create a secret with an auth token or a deploy key plus an extra workflow just to get this setup going.

This makes the setup much more complicated than it could be, and also much less reusable. Each repository where I want this I now have to copy both workflows and setup a new key and secret everytime.

This is essentially a combination of all well meant security measures, but they are making my CD very difficult right now.

[spydon]

I made a little bit more research, and publishing packages from a GitHub action is currently available in pretty much any language ecosystem that I can find:

• Docker
• Gradle
• Maven
• Node.js
• Rust
• Go
• Haskell
• C#
• Erlang/Elixir
• Python
• Ruby

@jonasfj @isoos are there any specific concerns that you have with enabling workflow_dispatch here?

And also; Happy new year! :sparkles:

[jonasfj]

@spydon, the idea being that with workflow_dispatch you can create a button in Github, and clicking this button will:

• increases my pubspec version
• stamps my changelog
• commits these changes
• creates a github release
• publishes to pub.dev

As outlined by @clragon in https://github.com/dart-lang/pub-dev/issues/7177#issuecomment-1879892823

I don't think that's a bad idea. Off the top of my head I don't think have any objections to enabling workflow_dispatch.

Though we might need to investigate further exactly who can trigger a workflow_dispatch (like we'd need some input for at-least a lightweight security review). And we'd need to build some tooling for all of the above, or at-least write some documentation.

I'm a little bit conservative about opening up features like this, because they are really hard to remove once we ship them. And currently focus is somewhere else.

Ideally, I'd want a workflow_dispatch action that creates a PR, and the PR creates a tag. But I don't think creating tags from PRs is possible today. At-least not without using a Github Action, which means that any tag-push Github Action won't run.

Using the flow above, you'll end up with git commits in the tag, that have not necessarily be landed on main/master branch (I guess the action could do that), but certainly the commits wouldn't have a review associated with them in a pull-request. I could imagine that being undesirable in some future where SLSA can attest to whether or not all changes making it into a release have code reviews. I don't think that's on SLSA roadmap yet, hehe, so that's just me thinking too far ahead.

Note: I'm not saying we shouldn't do this. Just we should make sure it what we really want. And that it's the right flow. And that maybe it's not super urgent, because we have workaround for people who want really complicated automation.

---

Crazy idea

What about creating a release preparation workflow, to be triggered with workflow_dispatch, which will:

• Bump version number in pubspec.yaml
• Add a section to CHANGELOG.md
• Commit changes
• File a pull-request with changes

In the pull-request comment it could include a link as follows:

https://github.com/<org>/<repo>/releases/new?tag=v<version>&target=main&title=<title>&body=<body>

Then all a human has to do is:

• Trigger the release preparation workflow using a manual workflow_dispatch.
• Review and merge the pull-request created.
• Click on the link to create a release, and hit "Publish Release" button.
  • This will create the tag
  • It will create a release
  • All fields can be pre-populated from query string.

Another variant of this could be to make the release preparation workflow create a "draft release". Again, you'd have to open it and click "Publish release", but everything else can be automated.

Just an idea. It also really depends on what flow you have.

[spydon]

Though we might need to investigate further exactly who can trigger a workflow_dispatch (like we'd need some input for at-least a lightweight security review). And we'd need to build some tooling for all of the above, or at-least write some documentation.

It will be the same people as can create releases today by creating tags, so what tooling etc would need to be written for this?

I'm a little bit conservative about opening up features like this, because they are really hard to remove once we ship them. And currently focus is somewhere else.

Makes sense, I think this is quite an important feature though.

Ideally, I'd want a workflow_dispatch action that creates a PR, and the PR creates a tag. But I don't think creating tags from PRs is possible today. At-least not without using a Github Action, which means that any tag-push Github Action won't run.

There is no problem creating tags from GitHub actions when a PR is closed, that is exactly what will be done with the melos action once workflow_dispatch is enabled. The problem GitHub actions are that they can't be triggered by tags created by other actions, hence the need for opening up for workflow_dispatch.

Using the flow above, you'll end up with git commits in the tag, that have not necessarily be landed on main/master branch (I guess the action could do that), but certainly the commits wouldn't have a review associated with them in a pull-request. I could imagine that being undesirable in some future where SLSA can attest to whether or not all changes making it into a release have code reviews. I don't think that's on SLSA roadmap yet, hehe, so that's just me thinking too far ahead.

No, the tags are created automatically upon merge of the release PR.

Crazy idea

The workflow will be even simpler than that:

• Trigger the release preparation workflow using a manual workflow_dispatch.
• Review and merge the pull-request created.
• Once the PR is merged another workflow will: - This will create the tag - It will create a pub release - It will create a GitHub release

[jonasfj]

The workflow will be even simpler than that:

• Trigger the release preparation workflow using a manual workflow_dispatch.
• Review and merge the pull-request created.
• Once the PR is merged another workflow will:
  • This will create the tag
  • It will create a pub release
  • It will create a GitHub release

So "another workflow" will publish to pub.dev, how is that triggered?

By merging the pull-request? Because that would imply publishing from a push event on master/main, and not workflow_dispatch.

And publishing from push to main/master is kind of something we'd like to avoid having people do. It's just too easy for someone to publish on every commit, which is a bad idea. Hmm, or maybe, I'm too worried about building footguns :rofl:

Makes sense, I think this is quite an important feature though.

Agreed, but the big feature with automated publishing is that you don't need credentials on your laptop or long-lived credentials in environment variables on Github Actions.

[spydon]

So "another workflow" will publish to pub.dev, how is that triggered?

You can trigger workflow_dispatch workflows from other workflows, which is what the melos action is set up to do.

By merging the pull-request? Because that would imply publishing from a push event on master/main, and not workflow_dispatch.

And publishing from push to main/master is kind of something we'd like to avoid having people do. It's just too easy for someone to publish on every commit, which is a bad idea. Hmm, or maybe, I'm too worried about building footguns 🤣

Understandable, I think since it is a bit complicated to build these workflows manually I think that most people will rely on pre-made actions like ours that sets this up for them and thus avoiding the footguns. It could even be undocumented that workflow_dispatch can be used, if that is a big worry.

Agreed, but the big feature with automated publishing is that you don't need credentials on your laptop or long-lived credentials in environment variables on Github Actions.

I think since virtually every other big language ecosystem supports this I think we would have heard of a lot these problems before if it would result in a big problem. :slightly_smiling_face: And also, since this is already possible to do through the Google ecosystem, it seems a bit strange that it would be restricted when doing it through GitHub, but not through Google?

[jonasfj]

It could even be undocumented that workflow_dispatch can be used, if that is a big worry.

Undocumented ways to publish things sounds a lot like a future security vulnerability :see_no_evil:

I suspect that it would have to be opt-in, with a checkbox on the "admin page" for each event that should be allowed to publish. Or maybe a radio button is better, no need to allow publishing from multiple events.

But I can see an argument for not being too worried about workflow_dispatch, it probably won't encourage people to make a tag and publish a release on every push to main.

Overall, I like the idea of allowing workflow_dispatch, because it means we can still require that the code is pushed to a tag. It just allows users to create workflows that automatically push the tag and then trigger a workflow_dispatch.

There is still an open question around how provenance will look for such workflows. How to even make provenance is not resolved yet, so yeah maybe we shouldn't block on it. But I'd at-least be curious to see what the JWT from Github OIDC on such jobs claims.

[jonasfj]

This is not a full design, let's reach consensus on what the right thing to do is before we start adding new settings.

[spydon]

Undocumented ways to publish things sounds a lot like a future security vulnerability 🙈

Agree, I was grasping for straws to get this in. :sweat_smile:

I suspect that it would have to be opt-in, with a checkbox on the "admin page" for each event that should be allowed to publish. Or maybe a radio button is better, no need to allow publishing from multiple events.

I would just allow it by default after enabling publishing from GitHub actions, I don't see how a logged in user with admin rights can be a security threat, because I guess that is your thinking? And also, this is already allowed through GCP, without any checkboxes, so I don't see why it couldn't be enabled for GitHub?

If we go with opt-in, I would make them checkboxes and not radio buttons, because you might want to both push tags manually and press a button in the UI at the same time.

[spydon]

@jonasfj any thoughts on not having any checkboxes at all? :smiley:

[jonasfj]

create an extra commit just with a tag that releases the version

You don't need an extra commit, you just need to push the tag.

Or create the tag through the releases UI on github.

Which can be done from an easy link, as outlined in "crazy idea" in https://github.com/dart-lang/pub-dev/issues/7177#issuecomment-1895918750

I actually suspect it might be worth prototyping something like that.

[spydon]

Then you have to manually create tags in the UI, as I explained here you can't trigger workflows from a tag created in another workflow. And to create each tag manually in the UI is even more effort than creating them from the terminal.

What I'm building is specifically for monorepos which are the ones that need this the most, so in the case of Flame for example you'd have to click ~30 links if you'd follow the structure in your "crazy idea" plan. Or for flutterfire for example I suspect it would be up to 50 links.

I don't understand what is blocking you from just having workflow_dispatch directly alongside push? Pub already support this flow through google cloud, I don't understand what the problem supporting it from GitHub as well, can you please explain because it feels like we're going in circles here.

[spydon]

Here's an example of how such a PR could look like: https://github.com/flame-engine/flame/pull/3044

[jonasfj]

I don't understand what is blocking you from just having workflow_dispatch directly alongside push?

Non-trivial changes generally trigger a formal launch process, which in-turn means that:

• A design document of some sort, but at-least a one-pager outlining what, why and various implications,
• A privacy review, if we storing new user-data (including something simple like settings),
• A security review, if we're tweaking security sensitive things like authentication / authorization.

Once I get to actually push the buttons, some of these can be trivial. Not all features involve storing new user-data. And many changes don't have security implications. This change has a non-trivial probability of having both security and privacy implications.

While this can take time, it's perhaps a good thing that we don't tweak authorization without having a fully conceived strategy.

Right now, I don't think there is bandwidth for driving the process. I have other stuff cooking, but figuring out what other triggers we should consider, and what the limitations of those might be, could get us a lot closer to a design doc.

So far I'm sort of reaching the conclusion that a lot of use-cases could be covered by dispatch_workflow.

So maybe we'll want to:

• Enable package authors to allow/disallow publishing from:
  • dispatch_workflow events,
  • push events,
• Enable package authors to disable publishing from command-line.

At-least that'd facilitate a lot of the use-cases here.

---

any thoughts on not having any checkboxes at all? 😃

I suspect it would be risky, given the two reasons below (a), and (b).

(a) New authorization mechanisms have to be opt-in

If someone read the documentation (http://dart.dev/go/automated-publishing) and enabled automated publishing, they may have a release process with security that relies on the documented properties. If we change the authorization model to allow new ways of publishing, then they may unwittingly have security vulnerabilities.

When it comes to authorization, I think it might be bad to allow something that wasn't allowed before. Without explicitly having people opt-in to this new "feature". Because while you might consider it a "feature" someone who doesn't use might consider it a vulnerability.

We didn't enable automated publishing from Github without opt-in. I doubt we can extend it without making this an opt-in feature either.

(b) You should only allow the required event types

Regardless, if whether you opt-in or not. I think that if you are only using workflow_dispatch for publishing. Then it would probably be preferable to disallow push. As was as disallowing publishing from commandline.

Because if you are designing a publishing workflow, you should want to prevent anyone from your team from accidentally publishing around the workflow -- either manually or by pushing a tag.

---

Here's an example of how such a PR could look like: https://github.com/flame-engine/flame/pull/3044

Do you publish all these packages in lock-step? If so I'd be curious why. Why not make a single package?

[spydon]

Thanks for the elaborate reply!

So maybe we'll want to:

Enable package authors to allow/disallow publishing from:
    dispatch_workflow events,
    push events,
Enable package authors to disable publishing from command-line.

At-least that'd facilitate a lot of the use-cases here.

I agree, that should cover all use-cases I can think of.

(a) New authorization mechanisms have to be opt-in

That makes sense.

(b) You should only allow the required event types

Because if you are designing a publishing workflow, you should want to prevent anyone from your team from accidentally publishing around the workflow -- either manually or by pushing a tag.

We can't know what types of processes the users have though, there might sometimes be a need for manual publishing, so I think that it should be possible to have all enabled if one wants to (i.e. checkboxes, not radio buttons).

Do you publish all these packages in lock-step?

Yes, the ones that have changes, they are published through melos publish.

If so I'd be curious why. Why not make a single package?

Because they are versioned differently and have different dependencies, a breaking change in one bridge package should not be regarded as a breaking change in all the packages for example. Usually Flame users depend on the core package and then possibly one or two of the bridge packages.

A design document of some sort, but at-least a one-pager outlining what, why and various implications,

Is there a template for this? Maybe I could help out with this and it could be a collaborative effort in a google doc? :)