microsoft
API Breaking change detection
This epic covers the tooling work required to perform automatic breaking change detection. It does not cover any repo or release process changes for its usage, which is instead tracked under #8007, although the two are heavily interconnected. While there are many existing tools that help manage required version bumps for individual PRs, none automate the detection and instead require the developer to determine the required version changes and tag the PR appropriately.
Goals
- Tool should be able to automatically detect if an API change produces a major or minor breaking change as defined by semver
- Tool should provide a high level of accuracy and should err on the side of false positives (over-detection -> unnecessary releases) rather than false negatives (under-detection -> missed releases)
- Tool should be integrated into PR validation to automatically notify developers of breaking changes
- Tool will be updated to support repo changes as covered in #8007
Non-goals
- Tool should not address non-API breaking changes, such as API behavior changes and document format changes.
Proposed methodology
Type compatibility in Typescript can catch major breaking changes in a lot of cases, but is fundamentally different from API compatibility. This (probably) makes it impossible to leverage the TS compiler itself to cover the full range of API breaks. Class type compatibility in particular has its own challenges in TS (e.g. private/protected methods are never comparable unless from the same parent class, which will not be the case when comparing between two versions of the same package). There are then two options:
- Implement the compatibility check ourselves (similar to implementing type compatibility ourselves)
- Convert classes into a representation that when compared for type compatibility will also uncover API breaks. We opt for the latter because it better leverages tools that already exist, but both require examination of the AST.
Work items
- [x] #7915
Improving coverage
March
- [x] #8029
- [x] #8038
April
- [ ] #8041
- [x] #8949
- [ ] #8950
Future
- [ ] #8037
- [ ] #8040
- [ ] #8042
- [ ] #8043
Integration
April
- [x] #8028
Future
- [ ] #8026
- [ ] #8027
Have we explored using https://api-extractor.com/ for this rather than rolling our own version? What other tools were considered?
we do use api extractor already, but it doesn't quite tell you if a change is breaking or not. just that the surface area changed, which can be ok in the case of some optional or new things
I'm curious what other repos do, is this really a problem that is unique to FF that we need to roll a heavy solution of our own like this?
In general, most of the tooling i've seen depends on the developer to annotate changes in some way. This is how beachball, semantic-version, and yarn2 all work. From those annotations it constructs the true release versions.
The goal of this tooling is to augment that process, but like you i worry about the weight of a solution like this