RFC: Amplify api pull command

[UnleashedMind]

Overview

Currently, if a GraphQL API is updated outside of the Amplify CLI, such as in the AppSync console or service API calls, there is no convenient way for the customers to pull down those updates to their local development environments. Based on customer feedbacks, such as these feature requests for a “pull” command, https://github.com/aws-amplify/amplify-cli/issues/174 https://github.com/aws-amplify/amplify-cli/issues/293 https://github.com/aws-amplify/amplify-cli/issues/1190 we are evaluating the “amplify api pull” command that aims to bridge this disconnection between the local development and the AppSync’s web console. This RFC defines the use case, analyzes the problems and options, and asks the community to give us feedback and suggestions on this command.

The use cases

The following are the steps of the most common workflow for the amplify api pull command:

1. Customer adds a new GraphQL API to his project (or updates an existing API) in the local development environment and pushes local development to the cloud $ amplify api add / update $ amplify push

2. Customer goes to the AppSync console and makes modifications on the GraphQL API, such as adding a new type, adding/removing a field from a type, or changing some of the resolver mappings. $ amplify api console

3. Customer comes back to the local development environment, pulls down the developments that happened in the AppSync console, and continues his local development. $ amplify api pull

There’s another use case that is associated with a “pull” command, team member collaborations using the Amplify CLI: Developer A pushes his local development to the cloud, and developer B needs to pull down the changes to his local development environment. This use case is addressed by the amplify env pull (https://aws-amplify.github.io/docs/cli/multienv#team-members-sharing-the-same-dev-backend) command, which is already implemented, and out of the scope of this document.

The amplify api pull command focuses on giving the customer the ability to combine local developments using Amplify CLI with the developments in the AppSync’s web console or any other methods outside of the Amplify CLI management.

Conversion between schemas

In developing a GraphQL API, the developer’s input to the Amplify CLI is not a regular graphql schema, but a schema annotated with graphql transformer directives. The annotated schema is called the “input schema document”. The Graphql Transformer first builds the annotated schema locally, transforms it into the API’s regular graphql schema, resolver mapping templates, and CloudFormation templates, etc.. Those build artifacts are the actual items used by the amplilfy push command to construct and update the GraphQL API in the cloud. When retrieving the latest updates from the he AppSync’s web console, the AppSync's public API methods, such as 'getIntrospectionSchema', ‘listTypes’ and ‘listResolvers’ etc., return the regular schema, the resolver mapping templates and other resources. They are equivalent to the above mentioned build artifacts. And currently there is no tool to reverse engineer, and re-construct the annotated “input schema document” from the build artifacts.

To summarize, the AppSync console and the local development are represented by two different formats:

• Local: annotated schema
• AppSync console: regular schema and other resources:

The hard problem is how to diff them and represent the diffs in the local development’s input format: the annotated schema.

The options

1. Reverse GraphQL Transformer’s build operation: Implement a full-fledged tool that reverses the GraphQL Transformer’s build operation; use it to construct the annotated schema from items retrieved from the AppSync service; and then use the constructed annotated schema to diff, resolve conflict and merge with the developer’s local “input schema document”. This would be the ideal solution if it is possible. However, it does not seem to be a trivial problem upon examining the current operations, and we would first need to refactor the GraphQL Transformer to make sure all its operations have reverse operations.

2. Best effort approach: Set the goal a little differently, from providing the pull command's functionalities in full, to that the cli will use its best effort to pull from the AppSync console. And if the cli cannot handle the situation, it will guide the customer to restore the project locally and in the cloud, so that she/he can continue to use the CLI in the project’s development. The ability of the “best effort” can be iteratively improved to include more and more use cases. Pro: Although the scope is limited, the partial implementation can be useful in certain scenarios, and it preserves the input format: annotated schema. Con: Can be confusing because “sometimes it works, sometimes it does not”; it can also be complicated to use when the cli tries to restore the project if it can not handle the discrepancies between the AppSync console and the local development.

3. Use regular schema The pull command will replace the annotated schema with the regular schema. Developments after the pull command will be directly on the regular schema, the resolver mapping templates and the CloudFormation templates, etc. New resources can still be set up by the GraphQL Transformer, for example, adding a new @model annotated type into the regular schema will set up the DynamoDB table etc.. Of course, the schema is now an annotated schema again. Pro: Easier to implement than the other options. Can pull down all the developments happened in the AppSync console, it is a full implementation of the pull functionalities. Con: If the customer starts with annotated schema, the execution of the pull command will change it to regular schema, it can be confusing. Development will involve both annotated schema and direct resource specifications, such as datasources and resolve mappings etc.. And there can be discrepancies and conflicts between themselves as the combined input documents, which introduces more complexities in the development.

Please comment with your thoughts and suggestions.

[hisham]

Option 2 seems best. It may be worth also to allow us to pass parameters to the pull command to target specific resolvers. Often we're just iterating on a custom resolver code and nothing else. That might simplify things.

I think when creating models and developing connections, using the annotated schema and doing a push will be faster than doing this stuff manually in the console and doing a pull.

Using the console however is useful when customizing resolver code and adding custom data sources that the graphql transformer does not yet support. So I think the pull command is most useful for custom resolvers and data sources that would need a custom nested stack anyways rather than trying to recreate an annotated graphql schema.

[jiajern]

Hi all, so what will be the work around currently? If I add an API using amplify add api and then later on when I want to change the schema, do I have to modify the schema and the graphql query that were generated at the beginning? Thanks in advance.

[tracycaruso]

1+

[YoanGJ]

First I'm very happy to know that this feature is seriously considered! I'm not using the CLI anymore because I can keep my work in sync.

I think that the first option is of course the best and if you're currently investigating this way it would be awesome to have an ETA of your work so we can have an idea to know if there's some hope to see this solution arriving between our hands anytime soon.

But personally I'm ready to wait longer without the CLI instead of having kind of a partial solution with option 2 & 3.

[phil-andrews]

This is the one big drawback I've been seeing when trying to run AppSync as a multi-client backend. There's no seamless way to modify in one client and then pull it into another. Seems to be a no-brainer that if the source API has changed that any client configured to it should be able to pull in the changes. Almost identical to git but without having to go through the process of setting up and maintaining a separate AppSync repo.

Vote = Option 1

[Genkilabs]

+1 Bump. This would be really great for me right now. ^_^

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because of inactivity. Please open a new issue if you are still encountering problems.

[Genkilabs]

@kaustavghosh06 Thanks for keeping this request alive.

[ambroserb3]

+1

[jesse-cape]

+1

[kenchoong]

+1 this should be done

[araker]

+1 for option 1. Or make it very clear when the discrepancies can't be handled in case of 2 or schema changes (annotated <--> regular) in case of 3.

[jkeczan]

+1 for option 1.

[tostangs]

+1 for option 1 if this feature is still being considered. Not sure why It's not already a feature since this could be huge for maintaining a single codebase for AWS infra. The lack of this feature will likely be the reason my company stops using this framework. This feature is a must and should be implemented throughout the entirety of the framework

[bpolonia]

+1

[volkanunsal]

Any progress?

[andystod]

+1

[baljanak]

+1

[bourkison]

+1

[drclohite]

I have tried using Amplify pull and it works seemingly without error. The hosted amplify and appsync apps exist and work as does my new environment, however when I install all the auth code my new pulled app works without error, but no authentication process appears.

index.js: `import React from 'react'; import ReactDOM from 'react-dom'; import Amplify from "aws-amplify"; import awsExports from "./aws-exports"; import './index.css'; import App from './App'; import reportWebVitals from './reportWebVitals'; Amplify.configure(awsExports);

app.js: import React, { useState,useEffect, useReducer, Fragment } from 'react'; import { API, graphqlOperation } from 'aws-amplify' //import { listTodos2 } from './graphql/queries' import logo from './logo.svg'; import './App.css'; import { withAuthenticator } from '@aws-amplify/ui-react' import Auth from "@aws-amplify/auth";`

function App() { `const [error, setError] = useState(null); const [username, setUsername] = useState(null); const showlog = 6; useEffect(() => { try { setError(null); Auth.currentAuthenticatedUser({ bypassCache: false // Optional, By default is false. If set to true, this call will send a request to Cognito to get the latest user data }).then(user => { if (showlog === 6) console.log('currentAuthenticatedUser username',user.username,'auth',user.attributes.sub, 'email',user.attributes.email) setUsername({'username':user.username, 'sub':user.attributes.sub, 'email':user.attributes.email}) }).catch(err => setError(err)); } catch (e) { setError(e); } }, []); return ( <div className="App"> <header className="App-header"> <img src={logo} className="App-logo" alt="logo" />

Edit src/App.js and save to reload.

<a className="App-link" href="https://reactjs.org" target="_blank" rel="noopener noreferrer" > Learn React ); }

export default withAuthenticator(App)`

Also, I cannot see how to import graphql queries and the various pages I have found on the web about amplify pull don't cover how this should be achieved. My original app contains a graphql directory with mutations, queries, schema, and subscriptions. Just what I need to develop in that app. This directory or a way of linking to those files in the 'mother' app does not appear in any obvious way and there does not seem to be any documentation on how this might be achieved or what this might look like online.

My original app, the mother app file as it were contains an import statement: import { getTodo } from './../../graphql/queries' what do I put in the pulled app? Also how do I get auth working.

aws-exports shows the correct aws_user_pools_id, aws_user_pools_web_client_id, and aws_appsync_graphqlEndpoint

[Rafcin]

@drclohite In the amplify console when you select a backend environment it should give you a quick copy and paste for amplify pull with a env param so you can pull that build. Often that works for me and then just run amplify codegen to generate your graphql queries and mutations. Not sure if that's the answer you need, I haven't kept up with this thread.

Auth can be really tricky sometimes. Often if Auth doesn't work properly I do amplify configure auth and I walk through the entire config (and it should have your existing options selected) and then just rapidly press enter and when it's done everything works. That's the way I fix auth when it says I'm missing my Google client id.

[drclohite]

I figured out that pull/Auth was working and works fine. That leaves me not knowing how to read my graphql queries... Any clues would be helpful. I did not understand what run amplify codegen means although I will continue to look. I have raised a support query and asked on a forum, the forum led me to realise Auth was there.

[drclohite]

it was codegen and the issue is resolved. Thank you

[jay-herrera]

+1

[feathars91]

+1

[seanim0920]

+1 This seems like a very basic feature that amplify should support. I'll have to reconsider using Amplify for my next project.

[jsonchen47]

bruh how is this not resolved yet