sofie-core
sofie-core copied to clipboard
nrkno
Sofie Stable API
The issue has been opened by SuperFly.tv on behalf of EVS Broadcast Equipment.
Motivation
As part of a project being developed by EVS there is a desire to control Sofie from an external system via a REST API.
Currently, Sofie auto-exposes some API methods, however, this auto-generated API comes with no stability guarantees nor any documentation. Therefore, we propose to develop a stable API that is documented by the OpenAPI standard and does not rely on exposing internal Sofie APIs.
Proposal
We propose a system whereby an OpenAPI schema is used to generate a REST API using the Koa framework which will call a new RestAPI meteor API. This new meteor API will follow the patterns established for the UserActionsAPI, ServerPlayoutAPI etc. APIs. The RestAPI will then make whatever calls into the existing worker APIs (e.g. the StudioJobs API) that are needed in order to fulfill the API request.
We believe this approach follows established patterns within Sofie, meaning that the code changes should be reasonably self-contained. It also establishes a decoupling point at which internal changes to Sofie's APIs can be handled so as not to modify the REST API, and vice-versa.
As the OpenAPI schema will be the source of truth for the behaviour of the API, a strong contract will be established between Sofie and users of its API. The API route handlers inside Core will be generated automatically from this definition, meaning that a published specification is guaranteed to be reflective of the actual implementation. This definition will also be used to publish an auto-generated typescript client library, so external applications are provided with a library that is guaranteed to work for a given version of the Sofie API. Further, the specification can be used to auto-generate an API section for the Sofie documentation.
Due to much of the API being auto-generated, we believe that the maintenance overhead for this work will be relatively low, as work should only be required either when the API definition changes, or when there is a change internal to Sofie that requires the RestAPI to modify how API requests translate to Sofie internal API calls. We believe that both of these categories of changes should be relatively small.
Scope
For the initial implementation, we propose to create an OpenAPI definition that covers the UserActionsAPI, following the current implementation as closely as possible, as well as methods to retrieve some stable-api versions of the RundownPlaylist, Rundown, Segment, Part, Piece, Studio and ShowStyle collections, which will again follow the existing collections structure but with internal-only fields removed - and with querying by Id being the only supported query mode in this initial work.
We will also set up the required build steps to auto-generate the API handler inside Sofie and an auto-generated Typescript client library. The server stub will then be connected to the RestAPI as described, forming a working first version of this stable API.
As part of this work, no existing API methods will be removed or modified in order to preserve compatibility with any existing systems currently consuming these APIs.
There is not an ETA on when this work would be completed as we need just parts of the User Actions API before an upcoming deadline. We may prioritize getting the 50% of this work we need done for that particular deadline, and revisit the rest after said deadline before opening a PR for review, but a draft PR may be opened as a work in progress.
Request For Comment
What we are looking for is:
- Suggestions / comments regarding this approach.
- Anything people would change about the User Actions API for a public-facing version (e.g. is there a parameter we should not have to pass in order to invoke a certain action via the REST API).
- Any indication of other parties currently investigating the same topic.
- A thumbs up for this idea in general.
Speaking from an NRK perspective:
Certainly do look at the work done by @Julusian over at https://github.com/nrkno/sofie-core/pull/772, since that will affect the structure of the ShowStyle and Studio. , the question then being if this stable REST API should expose the unprocessed/un-extended objects, the processed ones, or both? If so, how?
Also, if this is supposed to be a Stable API, we must ensure that we agree upon a clear policy on the both the supported range of legacy versions of API we support/how many releases must pass before we drop support for a legacy version of the Stable REST API.
In terms of stability, we're not proposing that the API be stable across all future versions of Sofie / compatible with many, many versions of Sofie.
Instead what is proposed is that the API has a semver-compatible version, that may change from one release to another or may not. But when the major version of the API changes, then we accept that's a breaking change and that can happen on any release of Sofie.
What we would like is for semver-compatible versions to redirect inside of Sofie, so a call to v1.0.0 of the API would redirect to v1.1.0 in a version of Sofie that is exposing v1.1.0 of the API, which should be possible to do automatically.
For breaking changes, a call to v1.0.0 on a Sofie that uses v2.0.0 of the API would be expected to return a sensible status code e.g. 501 to indicate the the requested API version is not supported.
If that's the level of stability we are talking about, then I don't see any particular red flags at the moment. Koa is already used in many parts of Sofie, so it feels natural to use it here as well.
One question I do have is, suppose that I have a change which requires a large re-implementation of a method. Once done it would give the same api as before, so would be semver compatible, but NRK is unable to commit the time to re-implement it.
Of course, we should be trying to re-implement the method as part of making the underlying change, but sometimes it wont be possible.
In that scenario, what should happen?
Should that be a semver major change, so that it can be dropped? Potentially only temporarily, before being reimplemented with the same api?
Yes this should result in a major version bump, hopefully a PR will be opened by someone still using that API to re-add the functionality, which would result in another major version bump.
There's also the more abstract concept of "behaviour changes" in semver, a behavioural change should be met with a major version bump. That's not as easy to define and will probably take us a few goes to get right.
e.g. (contrived example) Take now moves two places on a Tuesday afternoon.
Wouldn't re-adding it be a minor bump? Because it will be a 'new' method, and so won't break anything
My one concern is that it feels like we could end with many major version bumps like this. It should be expected that we will work on large features spread across multiple releases, so unless we are careful, this could result in a breaking change each release as we break the next api method.
#772 is a good example here.
That PR changes the shape of the Studio document.
The follow up PR will change how the settings ui views and sets those.
Then another PR will change blueprint migrations, in a very breaking way.
In this case, they wouldnt all need to be breaking, but lets assume they were. That means 3 releases in a row with semver breaking changes. Is that desirable?
Good point, re-adding would be a minor change.
In a perfect world, the version of the Studio document exposed via the API would be different to the version(s) used within Sofie, so would be constructed from what's available and only change in a breaking way if it's no longer possible to construct the API-version of the document from the internal structures in a meaningful way.
This is a point for discussion, what should the API version of Studio / RundownPlaylist etc. look like? Ideally we'd expose the minimal amount to be useful so as to reduce the chance of internal changes requiring a change to what's exposed for the API. However, I think for the case of #772 we will be expecting that to break any API work that's in progress/in place before that PR closes.
OK - how can we kick off a discussion on the shape of the API's objects? I suppose this is going to be the most fundamental work here and will shape the methods of the API as well. Does anyone have any suggestions on a collaborative system that would allow us to track suggestions and changes?
My suggestions are either:
- A draft PR containing some empty interface definitions, where we can start a thread for each interface.
- A Google Doc.
Draft PR: https://github.com/nrkno/sofie-core/pull/777 Document: https://docs.google.com/document/d/1UI1GjKIojnlU9S9tYFc42Id1Ns01rtQSxfQs_WAxsG0
Please request access to the document, ping me if I'm not responding to invites fast enough. If you could add which organisation you represent to the request message that'd also be a great help.
This is all of course very early and just a starting point, I've added some empty interfaces to the draft PR for us to have some discussions around and I've put some questions in the document to get conversations started. Please add any/all comments/suggestions!
We'll follow up with a draft OpenAPI schema to the PR soon-ish which will hopefully prompt discussions over what the shape of the API as a whole should be.
I have updated the draft PR #777 with what we have written for our purposes. This is all of course open to change, it just represents a working example that meets the needs we had in the short term.
This work includes both a rest API and what we call the "Live status gateway" which provides a representation of the current state of Sofie over a websocket connection. This was originally planned to be addressed separately but it's clear that the two APIs are trying to solve two halves of the same problem, as knowledge of the state of Sofie is required to be able to have a meaningful interaction via the rest API.
Some key files to look at:
meteor/server/api/rest/api.ts- The entry point for the rest API implementation. Currently this includes Koa routes that are hand-typed rather than being auto-generated - which we have not looked into due to time constraints - but should be representative of what typing a Koa route to a Sofie method call would look like.packages/live-status-gateway/api/asyncapi.yaml- The schema for the "Live status gateway" sockets API. Feedback on the objects we're proposing to expose - whether over this or another API - would be greatly appreciated.packages/openapi/api/actions.yaml- The schema for the rest API. It currently primarily exposes methods from the user actions API - enough for an external system to control Sofie through the execution of a playlist.
The "Live status gateway" in general needs review. Is having a separate gateway the right idea? Currently, the gateway is per-studio - much like any other gateway - is this right? Does it make sense to have a streaming API for the state but a rest API for interaction make sense? There's also likely an overlap with the work done for the Input Gateway here, which I personally would be very interested to hear about.
There will, hopefully, shortly be a demo application available that demonstrates how the use of two APIs like the ones proposed in the draft can be used to build an application that interacts with Sofie.
Here is a link to an application that uses these two APIs. It's purely an experiment, so this application should not be viewed as representative but instead demonstrative.
If you run up a Sofie and attach the live status gateway to a studio you can then start this application. On the webpage that the appliction will launch, tick the "studio" checkbox and you should get a list of available playlists. You can then click on one of the playlist and click "Load playlist" to activate the selected playlist. From there you should be able to view some state information, a list of adlibs that you can trigger, and some other playlist actions.