fitnesse
fitnesse copied to clipboard
unclebob
Protect against API breakage
In FitNesse 20160618 there are changes in signatures of some protected methods of fit fixtures - which might be affecting other products overriding them.
E.g. commit 9cd4e5b2befe3348529fad66d2fc6224b399441b renaming ColumnFixture::bind method to bindColumnHeadersToMethodsAndFields is breaking DbFit RowSetFixture.java#L67.
Obviously for DbFit it's not hard to align with the new method name, just I'm not sure if other projects are affected.
I had previously drafted a PR (not yet submitted) to propose getting basic DbFit integration tests into the FitNesse build (as discussed in DbFit issue 411 (https://github.com/dbfit/dbfit/issues/411)).
@amolenaar - any thoughts?
@unclebob: Any thoughts on the change in ColumnFixture API?
Personally I do not touch the Fit code. One tends to break stuff pretty easily and there are no proper extension points.
@MMatten: I think it would be good to do integration tests as early as possible. The fact that we only find out now is just a damn shame. How do you see this work? Add tests to FitNesse itself? Or can we trigger dependent builds? Should we move everything in one big repository?
@amolenaar My approach was to include some Java-only (memory-only DB instances) DbFit acceptance tests (HSQL and/or Derby) in the FitNesse Ant build. The latest DbFit release is available from the Sonatype central repo so it's pretty easy to set up.
I don't know how we could hook up builds running on different CI services but I guess it's possible via URLs.
The DbFit CI includes MySQL and PostgreSQL integration tests that get provisioned during the TravisCI builds.
I think the easy and clean option is to just add tests to the FitNesse Ant builds. @javornikolov any different view?
One question regarding integration tests is how to manage the versions of dependencies (would we check master branches, latest maven versions; which Java version to use /e.g. we may move DbFit to Java 8+ some day/, ...).
I think the easy and clean option is to just add tests to the FitNesse Ant builds.
It can be this way to kick it off quickly if it's acceptable for FitNesse. DbFit is just one of the FitNesse clients so a general solution perhaps may evolve into something more complicated (e.g. separate project or sub-project).
@javornikolov I was thinking only of testing the DbFit release version from the Central repo in the FitNesse build not dev DbFit versions (these don't get into Sonatype at the moment anyway).
I had originally configured the Ant build to run the DbFit tests when building FitNesse with JDK7 or 8, but it was only building with 6 anyway at that time. We'd have to maintain the Ant build in line with Java version changes to DbFit.
@amolenaar Does the FitNesse build run tests using Java 7 and 8 (after building in 7) or just 7? Would it be possible / would you consider running the tests in both?
@MMatten, FitNesse already switched to Gradle build so I guess it might be easier to plug some tests now.
One question regarding integration tests is how to manage the versions of dependencies
I have no clear idea on how to solve that. Now that things are built with Gradle, it would be easy to upload snapshot versions to Bintray. But I'm not sure how to trigger downstream builds (let alone figure out what the important downstream builds are). I suppose you want to check both against the latest released version and the HEAD version -- we could no breaking changes on purpose.
DbFit is just one of the FitNesse clients so a general solution perhaps may evolve into something more complicated (e.g. separate project or sub-project).
True, there are a couple of plugins around right now. All can potentially suffer from the same problem.
@amolenaar Does the FitNesse build run tests using Java 7 and 8 (after building in 7) or just 7? Would it be possible / would you consider running the tests in both?
Everything is built against Java 7. the repo https://github.com/fitnesse/fitnesse also builds on Travis CI with Java 7 and 8.
@MMatten, FitNesse already switched to Gradle build so I guess it might be easier to plug some tests now.
We can add some plugin integration tests in the repo. It would create a cyclic dependency. That's not really nice.
I took the liberty to change the title, since we're discussing how to prevent API breakage for plugins in the future.
A few quick thoughts (not validated):
Cyclic dependency for single repo should be possible to resolve e.g. as non-cyclic dependency between sub-projects. Yet, I admit - perhaps it's cleaner not to mix too much stuff into FitNesse main repo.
But I'm not sure how to trigger downstream builds (let alone figure out what the important downstream builds are).
Triggering a dependent build I can imagine can be handled via commit or maybe via Travis API.
I guess it can be single separate repo with integration tests for different dependents. Interested parties may contribute their own bits.
Another approach might be just comparing API changes vs previous version (I found following tool via a quick search: http://lvc.github.io/japi-compliance-checker/. I'm not sure if it's good, maybe there are better ones).
If we're looking at triggering dependant builds in other projects, then any thoughts on ownership/process for dealing with breakages?
E.g. a successful FitNesse build triggers a build of DbFit/master and then that fails.
What happens next? I guess either downstream projects raise issues on the FitNesse project (just like this) or FitNesse contributors need to subscribe to all downstream projects to monitor build statuses.
Cyclic dependency for single repo should be possible to resolve e.g. as non-cyclic dependency between sub-projects.
I'm not quite sure that you mean. Can you explain differently?
@unclebob: Any thoughts on the change in ColumnFixture API?
@amolenaar - for this specific case should we consider rolling back the recent breaking change or perhaps the "cat's already out of the bag" on this one and there could be new dependencies built by people using the current release version although this has only just happened.
Cyclic dependency for single repo should be possible to resolve e.g. as non-cyclic dependency between sub-projects.
I'm not quite sure that you mean. Can you explain differently?
I meant something similar to what we have in DbFit oracle project depends on core but not vice verse. The core jars may be build alone without building oracle part.
If we're looking at triggering dependant builds in other projects, then any thoughts on ownership/process for dealing with breakages?
E.g. a successful FitNesse build triggers a build of DbFit/master and then that fails.
What happens next? I guess either downstream projects raise issues on the FitNesse project (just like this) or FitNesse contributors need to subscribe to all downstream projects to monitor build statuses.
The dependent project may be single one as one of the ideas I proposed. I think what happens after breakage depends on the change itself:
- It may be a desired one. If agreed that it's something good - downstream projects may need to accommodate to it and make the relevant changes too. (This might be accompanied by major version number change - e.g. as per http://semver.org/. Or might be done in 2-phases: deprecate but keep old API together with new one for a while; and remove deprecated stuff later).
- It may be unintended one: then FitNesse maintainers can change or revert it to fix the incompatibility.
@mmatten: I think it's not needed to roll back anymore. (sorry for the late reply :/ )
Another approach might be just comparing API changes vs previous version (I found following tool via a quick search: http://lvc.github.io/japi-compliance-checker/. I'm not sure if it's good, maybe there are better ones).
I think it would help to have at least a baseline of which api's are supposed to be stable (e.g. the base interfaces for WikiPage, TestSystem, and I suppose quite some classes in the fit package ;) ). FitNesse has quite a few public classes, for various reasons, but only a subset should be of interest to plugin builders.