Evergreen ember release testing

[lukemelia]

It would be nice to be able to specify an ember-try/travis config that tests the latest patch version of all minor ember releases above a certain specified one. As a maintainer of several plugins, it is easy for them to slip into a state where they are testing a minor version, and ember-release, but missing a minor version or two in between.

[kategengler]

I agree that its tricky to keep them all up to date.

I originally went with the scenarios because I've had things I wanted to test against multiple versions that required other packages changing in lockstep (for instance needing to go to an older ember-data along with an older ember). Something else to note is that we'll likely add npm support soon so that you could go to older versions of addons in a scenario as well.

Given that, my first thoughts:

• Support various test reporter formats to report cumulative results or results per scenario. This will support using ember try:testall rather than a matrix.
• Own a concept of "allowed_failures" so that ember try:testall can be used instead of a build matrix. Which will allow:
• Have a "simple" mode of ember-try where it just changes ember versions, based on a semver string. It could also always do "beta" and "canary" as allowed failures by default. Currently planning to announce Ember Observer will support showing compatibility based on a key in package.json, like so:

  "ember-addon": {
    "versionCompatibility": {
      "ember": "beta || canary || > 1.12.0" 
    }
  }

We could use this string in ember-try to test every point release available that satisfies that semver string. We would maintain the existing style of ember-try config so that anybody that needs to have multiple packages changed within a scenario could do so.

Thoughts? Is this at all what you were imagining?

[lukemelia]

This sounds like an excellent solution @kategengler.

Moving away from the matrix is unfortunate because it is so easy to see what versions are failing and what is passing, but I understand the motivation and don't see another solution.

Let me know if you want some help testing.

[mike-north]

the build matrix is sort of designed to handle this sort of thing. I wonder if we could get ahold of some travis-ci folks and pick their brains about this. @stefanpenner -- any ideas?

[mike-north]

Another thought -- as @lukemelia pointed out, the main value of build matrices for our purposes is having a clear indicator of passes/fails on particular versions. If we could come up with another solution that would provide similar visibility, it would be easy to let them go.

[kategengler]

While the build matrix is designed to handle this sort of thing, not everybody uses travis-ci and it has the same problem @lukemelia was originally reported -- you need to manually update with each scenario/point release you want to cover.

With this proposal, you can keep doing what you're doing with the build matrix and manually configured scenarios, this would just be for anybody that wants the convenience of not having to specify each point release in ember-try.js and the build matrix and who only has to vary ember versions between scenarios.