Automated e2e testing and pushing version upgrades to consumers

[mockitoguy]

Producer-client lifecycle

• When working on release tools I constantly build the sample project with new version of tools. It would be great to streamline this so that when we run "build" in the release tools project we have confidence that stuff works.
• I frequently need to bump versions of tools in the client projects (mockito, example project) by creating PRs and merging them. That's an overhead I'd like to eliminate. In general, I'd like the clients to be on recent version of our tools.

Above are the problems of the challenges of the producer-client lifecycle. Let's figure out a set of plugins an features that will streamline evolution and adoption of new versions.

Suggested implementation

Let's try to identify a scalable solution that can evolve into a general-purspose feature that allows convenient evolution of versions of OSS libraries. Below is the result of our internal discussions with @mstachniuk and @wwilk:

1. In producer (mockito-release-tools) we can configure a list of “clients” we care about most. We would start with 2: (mockito + mockito-release-tools-example).
2. For each client we provide a way to locally run e2e test with that client. For example: “./gradlew e2eTest” will run all clients, “./gradlew e2eTests:mockito:e2eTest” will run only e2eTest for Mockito client. e2e test is a) shallow clone b) bump version of producer to locally built c) performRelease -Preleasing.dryRun
3. Running “build” includes running all “e2eTest” tasks. So if one of the clients is broken the build fails and there is no release on CI.
4. When CI job is successful and we publish new version of producer we automatically create PR in GitHub for each client. The PR contains the version bump of the producer. For some clients we may choose not to publish PRs, those clients will be used only for e2e testing.
5. Bonus: we can add a functionality that will make the CI job of the producer automatically merge the PR created for the client. We would use it for some clients, like “example" projects. We don’t want to use it for Mockito, the Mockito team will decide whether they want to merge or not. So the “wait for PR and automatically merge” functionality needs to be configurable per client.

Suggested solution would scale because

• we acknowledge that there are different clients: those that we only use for testing, friendly projects that we submit PRs with version bumps, finally projects we own and want to force push the change
• we move the building of the clients (or at least some of that 😉) into their own CI pipeline
• we don’t bypass GitHub pull request model with force pushing changes without PR (Mockito team would probably not agree to force push version bumps)
• we don’t bypass client team’s validations when pushing code to their project. If we push version bump directly and it causes client project build failure who is going to fix it? :)
• we would wrap the feature in a sweet little plugin that is reusable all over the world :DDD
• we have easy way to build/reproduce locally

Starting point

To get started, let's get a single client local e2e test (example project) hooked up in the release-tools project. For version bumping, we can use Mockito's pattern of keeping the version of tools in gradle.properties file.

Future

In future this could potentially turn into more general capability we would add to release tools. It would be great if clients of release automation could automatically validate if their changes work with selected open source projects on GitHub. For example, I would love if every change of Mockito was tested with some other GitHub projects like guava. This potentially can be extremely useful feature.

[mockitoguy]

@mstachniuk, how does this one sound? Check out the "future" section, this feature can turn out into something huge!!!

[mstachniuk]

It sounds great! It's a huge. Let's maybe starts from something simpler. New successful release of mockito-release-tools automatically bump version of tool in mockito-release-tools-example, make some small change and release it! @szczepiq What do you think?

[szpak]

👍

I've implemented automated E2E smoke testing in my gradle-nexus-staging-plugin and I feel much more relaxed releasing new versions. Having those testing automated is crucial to provide reliable Continuous Delivery.

[szpak]

It would be great if clients of release automation could automatically validate if their changes work with selected open source projects on GitHub.

@szczepiq Again, we share the vision of the future of CD :)

[mockitoguy]

@szpak, we share the vision, now it's time to start working together :) Join us and we'll be unstoppable! 4 of us cranking great release automation toolkit!!!

https://github.com/Codearte/gradle-nexus-staging-plugin is a great plugin. We could merge our efforts :)

[mockitoguy]

Let's maybe starts from something simpler. New successful release of mockito-release-tools automatically bump version of tool in mockito-release-tools-example, make some small change and release it! @szczepiq What do you think?

Sounds great! We'll have more example projects, for different use cases (for example: minimal configuration, based on defaults VS full configuration; with notable releases VS without notable releases; with bintray VS with just maven central). So it would be good if the impl was generic enough to handle more 'test' projects in the future :D

[mockitoguy]

@mstachniuk, assigned to you, feel free to unassign if you choose to work on something else.

[mockitoguy]

I updated the description with a suggested solution.

[mstachniuk]

Current status: Check my changes on ms branch feel free to continue if you wish.

For now E2ETestingPlugin makes a few things:

• clone mockito-release-tools-example to mockito-release-tools-example-pristine
• clone from *-pristine dir to *-work dir
• execute ./gradlew publishToMavenLocal testRelease ... in *-work dir

For testing purposes you need to apply E2E plugin in *.gradle file and try run runTestReleaseMockito-release-tools-example task