Improved changelog and release notes, human readable updates

[houmark]

** Which Category is your question related to? **

General, changelog.

** What AWS Services are you utilizing? **

Amplify.

In #2080 I raised concerns with the stability and maturity of the Amplify platform, and the quality of releases. One point was that changelogs are hard to keep up with, partly because changes happen across multiple repositories and partly because the changelog for Amplify CLI is sometimes not updated at the time of the release.

Promises were made to make it easier for developers who depend on this platform to be able to have a more clear changelog, so it's possible to keep track of what may have broken, as for me almost every release broke something.

Well, it seems those promises were empty and updating the changelog went from sparse, to completely not happening. Looking at the "main" changelog linked in #2080 the latest version mentioned in the changelog is 3.0.0. Since then, I believe 12-13 releases have happened.

Due to my bad history of updating with breaking changes that took cost heavily on productivity in my small team, I decided to stop updating regularly unless something was broken or if there was a feature out that we really needed. It's now been a month and we are on 3.0.0 still and the first sensible thing is to read the changelog to get up to speed on potentially breaking changes in relation to our platform and the features we use, but there's nothing to read in relation to the CLI?!? I mean, it cannot be anything has been changed on the CLI?

Is this serious?

Yes, I am aware that I can scout all 10+ related repositories and read those changelogs, and I do intend to do so, but you said back then, that you would work to aggregate this better to make it easier to keep an overview.

[jkeys-ecg-nmsu]

@houmark First, in general, I think it's a best practice to not upgrade any package unless you have a good reason -- vulnerability hotfixes, performance optimizations, developer and user UX enhancements. It sounds like you've already adopted this policy, but thanks for sharing -- your story might save another small team some major heartache in the future.

I agree that a global changelog would be extremely useful.

It would also be nice to see Amplify updates in the AWS Release Notes page. I realize that Amplify is FOSS and for that reason is not considered a traditional AWS product, but it would still be nice to see Amplify notes showing up in the release notes. As a fringe benefit for the Amplify team, that would probably increase adoption of the framework by AWS customers who are still unaware of the Amplify revolution.

[houmark]

Over the 20+ years I’ve been doing this I’ve had the best productivity results by doing often and small atomic updates and keeping a close eye on changelogs to be ready to tweak whatever may (partly) break. On larger releases I generally hold back a bit and keep an eye on issues for that repository before upgrading.

What I’ve noticed, and the reason for my initial “issue” on this, is that Amplify seems to have many small releases and each and every one of them is somewhat breaking for someone. There seems to be little consideration to backwards compatibility or migration help by the platform when things are changed.

Amplify is great. Maybe a revolution. But in the year of 2019 our largest and potentially most difficult task is to select the main platform(s) we marry when doing a new project. Amplify may be skipped by many simple because of not having a great track record in terms of stability, lack of documentation, or a combination.

I feel like there’s many teams involved but they don’t talk together much. Amplify needs 1-2 project managers that can bind the glue between the teams to try to prevent breaking stuff, keep things in sync and make sure changes are communicated to paying customers.

[houmark]

And I wouldn't call Amplify FOSS per definition. It's based on commercial AWS products. It's an untraditional product yes because it's the glue on many services already existing in AWS cloud, but I probably would not have used any of those products if Amplify didn't exist. I would have considered other similar platforms instead.

[RossWilliams]

@jkeys-ecg-nmsu there are vulnerability fixes since the 3.0.0 release, but @houmark might not know about them because the changelogs are sparse. I’d like to see CVEs used like the EKS team does when open source code supplied by AWS has vulnerabilities. This would tie in with my scanning tools a lot better than reading commits on master.

[jkeys-ecg-nmsu]

FWIW it seems like it would be low-hanging fruit for the Amplify team to add git hooks to their internal build processes that merge the local changelogs into a global changelog and automatically submit a pull request.

If the problem is literally we have multiple changelogs, I don't understand why the solution isn't to just parse the local logs and merge them into a global log on a per-push or temporal (weekly) basis.

[jkeys-ecg-nmsu]

Also, w.r.t. the FOSS issue, it would be nice to get some clarification from AWS and/or the Amplify team on how the Amplify ecosystem fits into the larger ecosystem of AWS services. For instance, enterprise support currently disowns support for the CLI, whereas they support Amplify Console which as far as I can tell has the CLI as a direct dependency.

[houmark]

While automatically generated changelogs would certainly be a step up from no changelogs, Amplify should manually maintain these and add some inline documentation and/or links to documentation on each release. Google does this pretty well, and they are more or less in the same situation with monorepos etc.

Over the weekend I've tried to update from 3.0.0 to 3.14.0 (latest release) but after hitting one bug in 3.14.0 I downgraded to 3.11.0 which should not have that issue (this seems to be a node issue with certain node versions and something they could probably not have catched easily). On 3.11.0 I hit #2519 which I am certain could have been prevented by better migration tools and/or some description on how to manually ensure that an upgraded Amplify project has the correct configuration manually. This seems to once again be an issue where CLI works mostly with push'es, but the Console does not - or sometimes. This in my experience seems to be a major ongoing issue in Amplify. You simple cannot trust that Amplify Console will be able to consistently deploy your backend without issues and sometimes you need to push through CLI to fix weird stuff.

Anyways, this is getting long and the peace I had for a good month while not upgrading was exchanged for weird issues that should not happen on a production platform like Amplify.

[SwaySway]

Closing this issues as the changelogs are being generated in the recent releases.

[houmark]

@SwaySway Changelogs may be generated now, but did you ever read them? The version 4.7.0 has exactly the same content as 4.6.0 which is almost the same as version 4.4.0. It's kinda hard to identify what's changed.

Or is this not the correct changelog to look at? https://github.com/aws-amplify/amplify-cli/blob/master/packages/amplify-cli/CHANGELOG.md

[RossWilliams]

It would be helpful if part of the PR review process included checking the PR message is an accurate and full description of the changes.

I needed to review CLI changes recently and went to the changelog. For my own use of understanding fixes and features, the changelog provides no more help than reviewing the commits directly. This likely goes back to the commit messages often being short, and the PR template format not being used (no issue numbers or descriptions are entered for many PRs).

To take an example, #2742 has a title that says "new features", but the description in the PR does not say what these features are, and there is no linked issue. The commit is 50+ files, but few of the line changes appear to be related to these new features. I had a very hard time understanding the changes I needed to be aware of as a tool user.

[kevcam4891]

@houmark I can't 2nd, 3rd, and 4th this enough. The changelogs in this repo are useless. I wish the team would take a harder-lined stance in terms of requiring more substantive commit messages. I'm currently on version 3.12 and want to upgrade to 4.x, and am having a very hard time getting an at-a-glance list of whats changed. Multiple versions contain the exact same message in terms of changes, etc. I'm excited about Amplify -- been using it now for about 6 months and see the promise, but its got a long way to go to become a solid, stable, developer friendly tool.

[swyxio]

i might recommend https://github.com/algolia/shipjs as a nicer release automation tool that still forces you to write better release notes

[houmark]

There's still not a useful changelog. Really hard to know if there are breaking changes before upgrading.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[houmark]

Changelog is still very little useful. Issues should not be closed until the changelog meets standards developers can expect it to meet.

[kitsunde]

https://github.com/aws-amplify/amplify-cli/releases/tag/v10.5.1 This is not helpful:

The only way I can figure out what changes seem to be to read through the whole diff https://github.com/aws-amplify/amplify-cli/compare/v10.5.0...v10.5.1#diff-b29553b2c09d058d9df908a83a27c38557a2d8bd35732c41120470e9c6552219

It's really concerning upgrading amplify-cli versions when it's also what manages things that aren't easy to just recover like Cognito. Please amplify team, take this up 🙏