Automate publishing process via CI

[mtrezza]

New Feature / Enhancement Checklist

• [x] I am not disclosing a vulnerability.
• [x] I am not just asking a question.
• [x] I have searched through existing issues.

Current Limitation

Currently, publishing a new version of the Parse Flutter SDK on pub.dev is done manually. This poses several issues:

• Inefficient, as manual steps are required
• Possible divergence of GH versions and pub.dev versions
• Separate permissions management, instead of managing publishing permission as part of GH team permissions

Feature / Enhancement Description

Automate publishing process via GitHub CLI, analogous to, for example publishing parse-server on npm.

TODOs:

• [x] Add auto-publishing to pub.dev (triggered on GitHub release) https://github.com/parse-community/Parse-SDK-Flutter/pull/719
• [x] Validate by creating a manual release (3.1.x or 3.2.0) -- version tag without V prefix!
• [ ] Add auto-release workflow (branch renaming, changelog-generation, etc)
• [ ] Validate by creating a automatic release via PR-merge

Example Use Case

n/a

Alternatives / Workarounds

n/a

3rd Party References

n/a

[mtrezza]

@RodrigoSMarques It would be great if you could have a look into this, since you seem to be most familiar with the pub.dev publishing process. I am of course available to assist from the GitHub CI side. I think a good starting point would be to look at the parse-server workflows, specifically, the release flow.

cc @phillwiggins

[fischerscode]

@mtrezza @RodrigoSMarques If you want, I could have a look at this, too.

[fischerscode]

Some more information as I'm now home:

Currently there is no recommended way to publish dart/Flutter packages to pub.dev automatically. The community uses the access and refresh token pub publish created at the first publish. Technically those tokens are not package but account bound and belong to the personal account. This meas, although they work for publishing, in theory they are a security risk, as someone with push access to this repo could extract the (GitHub) Secret containing those tokens and use them in order to authenticate as the user who created them. As an example: If @RodrigoSMarques would create the GitHub Secrets using his personal account, anybody who has push access to this repo could steal those and publish to his package. Please note: This is just theory and I don't think anybody would do such a thing (right?) esp. as the fake release could be easily reverted by the pub.dev team. But never the less, it is worth considering the potential risk.

The options I see:

1. Accept the risk and use the this action. (Publish CI as soon as possible)
2. Create a Google Account that has just access to those packages. (Nothing relevant could be stolen.)
3. Publish manually until an official way to publish using CI is added. See this issue (Personally I am using this way for a small package.)

Some thinks the automation should do:

• On pre-release: make sure to only publish prereleases. (Check version)
• On release: makes sure not to publish as a prerelease. (Check version)
• Release the dart package first and the flutter package second (might fail otherwise as flutter might/should depend on dart).

[mtrezza]

@fischerscode Thanks for the analysis.

Yes, for now we'll publish manually until we found a feasible way to automate this.

We should do a careful risk analysis regarding automated publishing and permission exposure. The risk is not only malicious intent by a single person, but maybe more accidental actions or compromised 3rd party accounts, because we cannot control how someone stores their own account credentials.

@RodrigoSMarques Do you have a next release planned, where we can test how we coordinate a manual release?

[Mabenan]

Maybe i can give some new information on stealing secrets from github actions as i already tried this myself to get a lost key back. First of it is not so easy as every secret used in github actions can not be simply printed. I only could get by echo it to a txt file and upload this text file somewhere. Maybe we there are ways of securing the access to the secrets by restricting the environment to branches (or maybe cleanup and restrict the branch creation all together) and then restrict the merging to these branches.

[mtrezza]

We already use GitHub secrets for NPM and docker publishing token in other repos; we can assume the feature is safe as that is exactly what it's designed for. As I understand it, secrets are only accessible on the original repo branches, not on fork branches or PRs. We just have to be careful about what we merge, but that we have to be anyway.

That is one of the reasons we changed the permission model in our repos and are now giving only a select few people write access to the repositories, where before, every org member had write access.

[RodrigoSMarques]

Hi @mtrezza.

How can I contact you so we can talk about releasing new versions of the package?

Since Phil left the project we haven't published any updates, because I don't know how the process was defined.

I believe that as the package is now publisher verified, you could invite me to publish the version.

I await your contact.

[mtrezza]

Maybe we can already set up the CI to automate the publishing in the future? Do you want to look into that and open a PR for a GitHub workflow? Feel free to contact me on Slack.

[RodrigoSMarques]

Hi @mtrezza . I was away for vacation.

I don't know if there is an option for automated deployment.

My sugestion. Do the update manually and then try to automate.

We have several bugs to be updated and as I have other projects, time is limited to check before the update.

My sugestion. Do the update manually and then try to automate.

We have several bugs to be updated and as I have other projects, time is limited to check before the update.

In this issue we have the analysis about the deployment automation.

[mtrezza]

Sure, @RodrigoSMarques I think we could give you temporary maintainer access if you would be willing to publish a release? Then we have more time to figure out how to automate this.

[mtrezza]

@RodrigoSMarques Friendly ping in case you overlooked my previous comment.

[RodrigoSMarques]

riendly ping in case you overlooked my previous comment.

Good Morning. I did not forget. I'm in a period of a big holiday here in my region in Brazil.

Add my email (my github username @gmail.com) to pub.dev and I'll be able to publish the new version of the package.

He was involved in several personal projects.

I will try to get back to support for this package.

[RodrigoSMarques]

@mtrezza Friendly ping in case you overlooked my previous comment.

[mtrezza]

I should be able to give you permissions later today, I'll ping you here.

[mtrezza]

@RodrigoSMarques you've been granted publisher permission, please let me know if it works and thanks for your help to publish manually.

To hopefully gather more interest in automating publishing, I've added a bounty label; @fischerscode you wrote a comprehensive analysis, would you want to take look and open a PR?

[fischerscode]

@mtrezza I'll have a look at it. Do you think it would be best to publish to pub.dev once a new github release has been created or should we consider automatic github releases as well?

We could consider creating automatic patches to update the dependencies. Also, we could create releases automatically once a pr has been merged to eg. master. But I would have to research if there are any best practices.

CC @RodrigoSMarques

[mtrezza]

Amazing @fischerscode, we should add auto-release to this SDK as we are moving all repositories to auto-release. That means on each commit (depending on the commit message), a release will be automatically published.

The auto-release mechanism is essentially the same as in the Parse JS SDK, but it comes with its own complexities, so I would break it into two PRs:

1. Automate pub.dev publishing when a new GH release is published: we still need to manually add a changelog entry with every PR and determine the next version number based on the changelog entries.
2. Move to auto-release: it should be easy to refactor the CI workflow from step (1); if you are interested you can do that too based on how we already do it in the Parse JS SDK and I'll be on standby for any questions, otherwise I can take over this part; for auto-release we use semantic-release across all repos, also for non-JS packages, see for example the Parse Android SDK.

[fischerscode]

@mtrezza I've created PR #719. It addresses the first point. Pub.dev does not support token authentication at the moment, as a result, the action requires the refresh and access token (OAuth2) of a Google account, that can publish. I would suggest creating a bot account, granting it publish rights (pub.dev), use it to log in to google using pub and then create the secrets PUBLISH_OAUTH_ACCESS_TOKEN, PUBLISH_OAUTH_REFRESH_TOKEN based on credentials.json.

https://github.com/k-paxian/dart-package-publisher#accesstoken

I assume you are one of the administrators of the publisher. So I think it would be best, if you would create the bot account.

[fischerscode]

2. if you are interested you can do that too based on how we already do it in the Parse JS SDK and I'll be on standby for any questions

I could do that, but (as you wrote) we should focus on Nr. 1 first.

[mtrezza]

@fischerscode That's amazing and amazingly quick! I will create a Google account for that and add the secrets so maybe we can even try that out with our next publishing.

@RodrigoSMarques You may want to wait with manual publishing, so we try this out?

[RodrigoSMarques]

@fischerscode That's amazing and amazingly quick! I will create a Google account for that and add the secrets so maybe we can even try that out with our next publishing.

@RodrigoSMarques You may want to wait with manual publishing, so we try this out?

We can wait to test.

Who was having problems, now has a version to continue through the Fork

[parse-github-assistant[bot]]

Thanks for opening this issue!

• 🎉 We are excited about your ideas for improvement!

[mtrezza]

Do I understand it correctly, that these 2 packages (dart, flutter) will always have the same version number?

[fischerscode]

For now, they always had. But that's just how we did it.

We could treat both as independent. (Dart package goes through the release cycle and the flutter one just uses the dart one as a dependency. Similar to dart vs flutter (The software projects.))

[mtrezza]

Is that dependency opaque to the developer or do they have to manually add it and choose the version number of the dart dependency when using the flutter sdk?

[fischerscode]

Currently, they can just add parse_server_sdk_flutter: ^3.1.0. parse_server_sdk: ^3.1.0 is then automatically added, since it is a dependency of parse_server_sdk_flutter.

I we would release the packages independently (different versions) the developer would still be able to just add parse_server_sdk_flutter. But which version of parse_server_sdk is actually used might be a bit less obvious, but still, it could be checked by using the cli tools or checking pubspec.lock.

[mtrezza]

Got it, I'm asking for release automation so we know our constraints when designing the mechanism.

[fischerscode]

This might be the tool we are searching for the auto-release workflow.

[fischerscode]

Just looked a little bit more into melos... It's definitely the tool for us. It even updates version/changelog based on git commits. I might go over some of the testing suff and use it, since it handles the local path automatically and therefore makes the actions easier to read/maintain.

...but that's a project for tomorrow. (https://melos.invertase.dev/)

[mtrezza]

If possible I'd prefer to use semantic-release, it also supports monorepo. It is the tool of choice for us, currently used in almost all other repos and we try to keep our tool set as small as possible across Parse Org.