OpenAPI spec for the REST API

[taras]

Context

An organization that I'm working with is considering adopting Backstage as their developer portal. One of the hard(ish) requirements that they have is that all services that we consume provide an API specification. This requirement exists because the infrastructure team needs to automate communication with services. If a specification is available, then they can use code generators to generate API clients for these services.

Feature Suggestion

It would be very helpful if there was a way to ensure that services integrated into Backstage come automatically with specifications. .NET, Rails & FastAPI all provide a mechanism that automatically generates an OpenAPI definition from route definition. I'm not aware of any tools that work like this in Node.js world.

Possible Implementation

express-openapi is a middleware that generates some specification from routes. It doesn't produce complete specifications - I don't even know if this is possible.

This request could also be satisfied by a GraphQL API that wraps services that Backstages consumes. The only consideration is making sure that the design of the APIs covers plugin services as much as possible.

[andrewthauer]

I ran across this project the other day that could be of some use when using a code first approach: https://github.com/bee-travels/openapi-comment-parser.

However, I agree that using annotations or a middleware would be more preferable.

[stefanalund]

Does it have to be automated from the beginning, or could we start with a manually edited file?

[taras]

It doesn't have to be automated. Having an OpenAPI spec for services included out of the box could be a good start.

[stefanalund]

@taras wanna take a stab at it?

[taras]

I wouldn't mind but we need a way to ensure that they stay current. Any ideas?

[kaos]

I think it looks like express-openapi indeed produces a complete specification, although you will have to write the spec for each route manually. The upside to that vs a completely manual spec on the side is that you get the spec closer to the implementation. You may also get validation middlewares for your routes (both for request and response data) amongst others, if desired.. ;)

[andrewthauer]

I tried out https://github.com/bee-travels/openapi-comment-parser recently and it will generate the openapi spec through JSDoc comments. It's still manual, but the spec ends up being right right above the code. I haven't tried it, but I suspect you could use some of the existing openapi tools like validation in conjunction with this approach.

[kaos]

looked a bit more at express-openapi, and I must say that I was stumped to learn that you had to put your implementation along with the spec.. I don't fancy the idea that the spec drives/dictates the implementation, it should be the other way around (the spec reflects the registered routes)

[taras]

@kaos I agree - spec and implementation should be separate.

We've been experimenting with this inside of our Backstage instance for integrating with our platform. We've been able to decouple the frontend from backend development by using OpenAPI spec in the middle. We generate a client with autorest and use Prism to mock service while in development. We're going to put it through the grind internally and then make an RFC when we're happy with the workflow.

[kaos]

Thanks for the update, looking forward to it :)

[freben]

Just bumping this to say that there's definite interest in this still! :)

[OrkoHunter]

Hey @taras,

We're going to put it through the grind internally and then make an RFC when we're happy with the workflow.

Just curious if there is anything you'd like to share on your experience so far on this setup that you mention!

[taras]

@OrkoHunter yep, the setup that we have right now is this,

1. each service that we're consuming in Backstage has an OpenAPI spec
2. we generate clients for these services with autorest
3. the generated code is wrapped in a use*API hook which uses the Backstage's ref API
4. we use the use*API hooks in Backstage
5. we use Prism to create a mock server in development - this allow us to build UI before backend microservice is implemented

This is pretty spiky at the moment because Autorest is quite heavy. We could create a leaner generator that generates hooks automatically and eliminate glue. Prism is also not exactly what we want to because it allows mocking by matching requests but not a high fidelity simulation. There is an issue to make Prism more realistic.

I would like to see and contribute to is having OpenAPI spec for each service in Backstage. Then generate clients wether at built time or runtime from the OpenAPI spec.

[alexcurtin]

@taras I'm very interested in seeing how you generated the OpenAPI spec for the catalog API, do you mind sharing your implementation somewhere? Can you push up a branch showcasing what you've built so far? Our end goal is to be able to build a catalog-client from the OpenAPI spec for the /api/catalog endpoint. Thanks!

[taras]

👋 @alexcurtin we haven't integrated with the catalog so far. We only used the clients that we created to integrate services that already have OpenAPI specs into Backstage. I might have an opportunity to do it this week, but I'm trying to figure out how to validate the implementation.

[taras]

I'm looking at using json-schema-to-openapi-schema to convert at least a part of https://github.com/backstage/backstage/blob/master/packages/catalog-model/src/schema/kinds/API.v1alpha1.schema.json to YAML. From there, I'm going to try to create an OpenAPI spect that includes the routes and references the generated files.

[freben]

Nice. Yeah if we could reduce the duplication of defining the entity shape it'd be nice. But why API? I'd have expected Entity instead, since that's the actual return type. Unless you wanna get fancy and do oneOf each of the specific kinds, but the value of that is probably very limited especially since it won't account for custom kinds etc

[taras]

But why API? I'd have expected Entity instead, since that's the actual return type.

You're right. I referenced the wrong file. It was the last file opened in my tab instead of the proper file :) https://github.com/backstage/backstage/blob/master/packages/catalog-model/src/schema/Entity.schema.json is the right file.

[taras]

@freben I started creating a @backstage/openapi package. It'll contain the script and generated YAML files. Does that sound alright?

[freben]

Uh, I am not sure. @Rugvip @benjdlambert @jhaals any input on that?

Gut feeling says that the catalog API, for one, is a thing of its own and should go in the @backstage/catalog-client package with the other client functionality. And that an overall mega-api-definition for a backend, rather than individual plugins, might be of limited use.

[freben]

Regarding the linked ticket about refs, input is welcome. I have run into annoying situations where ajv and other consumers treat refs in (sometimes subtly) different ways. Some like paths, some like identifiers, ...

We could update the ref usage but it would have to fit at least ajv and openapi at the same time.

[taras]

We could update the ref usage but it would have to fit at least ajv and openapi at the same time.

I need to customize dereferencing to support identifiers. json-schema-to-openapi-schema also needs to be changed to support draft7. The rest of the conversion mechanism is quite simple so I might end up rewritinging the whole library.

And that an overall mega-api-definition for a backend, rather than individual plugins, might be of limited use.

Yeah, this doesn't seem great. I'm in doing this in part to see if I can use the OpenAPI spec to generate a frontend client. We also want to make sure that anytime that schema changes, we regenerate the OpenAPI spec.

[taras]

Could it be useful to hand craft the api schema first, and add an integration test that builds a js client from the schema and performs calls against a running backend?

That was my original plan, but I didn't know how to validate my hand crafted OpenAPI spec, that's why I went down the path of generating from schema file. I started researching how I could generate a client and I came across something that might be interesting.

Have you seen Optic? It monitors test traffic to identify endpoints. It can generate OpenAPI spec from information that's entered in their UI. It can on PRs to show any changes to those contracts.

I'm going to give it a spin over the weekend.

[acunniffe]

Have you seen Optic? It monitors test traffic to identify endpoints. It can generate OpenAPI spec from information that's entered in their UI. It can on PRs to show any changes to those contracts.

👋 Optic maintainer here -- this seems really useful and I'm happy to help out @taras and the community with some docs or guidelines for using Optic to document the REST API.

[taras]

@acunniffe 👋 I just got optic setup to start the backend and work as a proxy. It did exactly what I need it to - immediately.

[taras]

@freben I want to make sure that I understand what would be beneficial at this stage. I'm going to describe some user stories that we want to be able to support.

1. As a developer building a portal with Backstage, I want to see the documentation for REST APIs provided by Backstage core.
2. As a contributor to a Backstage project, I want to verify that I did not change the implementation of a backend plugin in a way that breaks a documented API contract.
3. As a developer building a portal with Backstage, I want to be able to easily integrate an existing OpenAPI documented service into Backstage with all of the necessary TypeScript/React Hooks glue.

Did I get these right? Are there other use cases that I'm missing?

For 1, as long as we have an OpenAPI spec that is current and accurate, we can use it to generate documentation. For 2, we need some automated tests that make calls from the browser to the backend to ensure that the client and backend stay in sync - the contract can be verified by observing these tests For 3, it would be nice if catalog-client was generated using this tool.

Thoughts?

[freben]

The main use case I had in mind was

• As an engineer at our company, I want to make an integration with the Backstage catalog from other non-Backstage services written in other languages, and would like to automatically generate clients for it

This is a HUGE use case for us. The catalog powers, or at least informs, so much more than just Backstage itself.

And as a result of that, and to get a good dogfooding going, I was considering exploring whether catalog-client indeed also could make sense to auto generate. As well as exploring generating express stubs for the server side from it - in a schema-first fashion.

But I don't know the viability of all that yet

[taras]

I don't know either, but I want to figure out how to make it happen.

@freben how attached are you to the JSON Schema implementation that we have today? If there was another way to perform that validation, would you be open to it? I just want to understand what can be changed.

[taras]

I've been keeping my eye on https://github.com/aws/jsii/ for a while. This could be used to generate clients in different languages.

Ok, I'm thinking that the first step could be to write a few e2e tests that use the catalog-client to interact with the catalog backend. We can use these requests to verify payloads that are sent to the backend API. Then use a proxy to inject a verification mechanism between the client and backend. The proxy will verify that the requests and responses conform to the spec.

Once we have this, then we can figure out how to proceed. How does this sound?

[jvilimek]

Hi guys. We are currently at the early stage of the Backstage implementation. And any such OpenAPI definitions and/or swagger help page would be really helpfull for us to see, what kind of API backstage and its plugins provide! Any news regarding this?