Using apiId requires authenticationType and schema.graphql to exist

[njeirath]

I'm attempting to use the apiId to attach resolvers to an existing API but am getting errors that it still requires that authenticationType and schema.graphql must exist. I may just be misunderstanding or misusing the new apiId but figured I'd ask the question.

Background

I have a backend serverless project that deploys my Appsync schema and a number of datasources/resolvers/functions. What I'd like to do is setup a new service say service1 that can deploy datasources/resolvers/functions related to service1 but tie into the existing schema deployed by backend.

When I attempted to use apiId only (in addition to my datasources/resolvers/functions) I get the error:

AppSync Plugin: Error: appSync property `authenticationType` is missing or invalid.

I added authenticationType: API_KEY but then it complains about not being able to find schema.graphql. I created an empty one to see if that would work but then CloudFormation complained that the schema was too short (requires a minimum of 1 character). Finally I tried moving just the parts of the schema that service1 would be deploying into my schema.graphql file but deploying that overwrote the schema that was already there rather than merging the schemas.

Questions

What I'm curious about then is:

1. I was able to get around having authenticationType required when using apiId by just setting it to API_KEY but I did notice that in the CloudFormation template that was generated it included an AWS::AppSync::ApiKey resource. While having that extra API key is not a huge issue for my use case, I assume it'd be better if authenticationType was optional when specifying apiId.
2. The bigger issue for me was having to specify schema.graphql as there doesn't appear to be any way to tell the plugin to simply use the schema that already exists or, ideally, to merge the schema for service1 in with the existing schema.

Again I may be just misusing/misunderstanding the purpose of the apiId configuration so any help would be greatly appreciated.

[bboure]

Hi @njeirath The apiId field is meant to deploy move an existing API to serverless. It will completely replace all resources of the existing API with those defined in the yml file. The error about the missing auth is definitely a bug, since the auth method will be retained by the existing API. The error about the schema is correct. You still need a schema that will replace the existing one completely.

I I understand correctly what you are trying to achieve is that you want to split your API into several stacks? The goal is probably to have sort of micro-services that live in the same API?

While it might work (if we removed the constraint on the schema existence), I would not feel confident deploying anything to production with it. The main goal of that feature is strictly to allow re-using existing API (in order to keep the url).

What you could try in the meantime is duplicating the full schema in both stacks and see how that goes, as a test.

[njeirath]

Thanks for the clarification @bboure

I I understand correctly what you are trying to achieve is that you want to split your API into several stacks? The goal is probably to have sort of micro-services that live in the same API?

You're correct, I'm trying to split my API out into micro-services that are fronted by a common API.

What I had been doing was my backend project would deploy my schema and then in my micro-services I was manually specifying my AWS::AppSync::DataSource and AWS::AppSync::Resolver resources to tie into the appropriate points in my existing schema (done purely with serverless and not using this plugin).

My hope was with the new apiId config that I'd be able to move away from having to manually define those resources and let the plugin do the heavy lifting but it makes sense that is not the intended purpose.

What you could try in the meantime is duplicating the full schema in both stacks and see how that goes, as a test.

I can definitely try duplicating the schema however I don't think it'll be a sustainable solution for us long term since keeping the schemas synchronized will become problematic.

While it might work (if we removed the constraint on the schema existence), I would not feel confident deploying anything to production with it.

Out of curiosity could you explain what your concerns would be about removing the schema existence constraint?

[bboure]

I can definitely try duplicating the schema however I don't think it'll be a sustainable solution for us long term since keeping the schemas synchronized will become problematic.

Yes, I can see that. I was just suggesting it as a workaround/testing solution

Out of curiosity could you explain what your concerns would be about removing the schema existence constraint?

I have no concern with removing that constraint. I am just saying that, in general, my feeling is that it should not be done as it sounds like a hack. It's not the purpose of this plugin, nor the apiId field to do that. The apiId is already sort of a hack used as an escape hatch for re-using existing APIs.

In fact, what you really want is Schema Federation but that is not supported by AppSync, unfortunately.

I'd suggest you to test the workaround I suggested and see if that works for you. If that does, we can debate whether removing or not the constraint. I am not against it.

Maybe @anastyn can bring some ideas here too

[njeirath]

In fact, what you really want is Schema Federation but that is not supported by AppSync, unfortunately.

100% agree here

I'd suggest you to test the workaround I suggested and see if that works for you. If that does, we can debate whether removing or not the constraint. I am not against it.

Understood, your suggested workaround to copy the existing schema into the service did work, I was able to deploy the service and it's datasource and resolvers attached to the schema as expected while also leaving the existing resolvers in place. Subsequently removing the service restored AppSync to the state it was in before the service deploy; it only removed the service's resolvers/datasources and left the schema and other resources intact.

This suggests the schema exists constraint COULD be removed however I agree it's more a hack and understand the reluctance to do so.

[anastyn]

Actually, I'm using the new apiId property in a similar way currently. Our schema grew too big and we decided to create multiple physical components (sort of microservices) and move parts of the schema and other resources into those components to be able to deploy and test them separately.

According to the GraphQL ideology, it is better to have one big schema sharing a single endpoint instead of having multiple small endpoints (+ AppSync limitation on the number of APIs). But the AppSync doesn't support anything like that so I've implemented it in multiple schemas and added a process to merge them before it starts the deploy.

Serverless has support for "setup" attribute which is analyzed before the plugin is executed. In this setup method, I've added a script to introspect the schema of the existing API, get the existing resolvers, and get the datasources. Then merge all of those with the ones in the current service, and then deploy them back.

So each service still has to have a valid schema, but it can have different types and queries serving different clients. The schema is merged with other schemas via graphql tools. I don't know if there is a point to have a service without a schema, the resolvers and data sources have to be connected to something, and that something has to be in the same service. We, for example, have separate data sources, resolvers, and types for different API clients, and we organize them into multiple services. One service has all data sources, types, resolvers for one client, another service for a different client. And they all are merged together into one big API, but can be tested individually.

Currently, I've tested 2 ways of getting information about the deployed resources:

1. Use AWS SDK to get them from AppSync. In this case, AppSync is the source of truth for the deployed resources. Something happens to AppSync API you lose the state.
2. Use S3. S3 is the source of truth for the deployed resources, plus we create backups after each deployment, and data is stored in multiple AZs. When the first project is deployed the resources are uploaded to S3 and to AppSync. After that, all subsequent deployments from other services are also uploaded to S3 and are merged with the existing resources. Each service has its own prefix on S3 and there is also a single merge result that is updated every time something is deployed. Plus every time we deploy we create a backup. If something happens to AppSync we always have the state of the API in S3 and can return everything to the way it was before the problem appeared.

What's cool about it is that every service can be tested and deployed individually without deploying the rest. And when we ready we can deploy it to one of the "shared" environments like uat or prod.

But it's still new and being tested.

[bboure]

I don't know if there is a point to have a service without a schema, the resolvers and data sources have to be connected to something, and that something has to be in the same service.

👍 I think @anastyn has a point here.

I think there is material for discussion here. I don't think AppSync will introduce federation any time soon.

With that being said, any solution that we can find to workaround that missing feature will always be somewhat "hacky". AppSync requires at least your full schema de be deployed. As I understand, you might be able to skip deploying some resolvers (if they are attached to a separate stack) and they would stay connected to your API. But who knows if this is going to stay that way? If AWS changes this behaviour, they will probably do it without notice. All your stacks will break. 😞

I totally understand the need for "microservices" and why would want to do that but I would still strongly discourage it for now. AppSync API should probably still be a big monolith.

Of course, that's just my opinion.

[njeirath]

AppSync requires at least your full schema de be deployed.

Agreed if the schema is being deployed at all. The approach we were taking was to have the entire schema defined as a monolith in a sort of parent backend project and this would be deployed once. Each microservice then deploys ONLY the resolvers and datasources they're responsible for attached to the appropriate places in the schema. I've used this approach on a couple of microservices so far by defining my resolver/datasource cloudformation resources manually and haven't run into any issues yet. It does require some coordination when making schema changes however for our situation I'm the only one working on the API so I know who's responsible if something breaks.

I do agree this is a hacky solution but, as a small team, it does give us a fairly lightweight workaround to AppSync not supporting federation without having to implement a schema merging solution which I can definitely see being beneficial for a larger team.

[anastyn]

Addressed the first concern about authenticationType in the following pull request: https://github.com/sid88in/serverless-appsync-plugin/pull/396