[DataStore] Optional fields related to another table using @belongsTo directive are throwing errors

[srimalaya]

Describe the bug

This is the schema that I am using:

GraphQL Schema

type Blog @model @auth(rules: [{allow: public}]) {
  id: ID!
  name: String!
  customs: [MyCustomModel]
  notes: [String]
  post: [Post] @hasMany(indexName: "postByBlog", fields: ["id"])
}

type Post @model @auth(rules: [{allow: public}]) {
  id: ID!
  name: String!
  blog_id: ID @index(name: "postByBlog")
  random_id: String @index(name: "byRandom")
  blog: Blog @belongsTo(fields: ["blog_id"])
}

type MyCustomModel {
  id: ID!
  name: String!
  desc: String
  children: [MyNestedModel]
}

type MyNestedModel {
  id: ID!
  nestedName: String!
  notes: [String]
}

I already have this schema running and working perfectly on a test app for Android and JS. Now, I am trying to test DataStore on iOS (Swift + SwiftUI). With the above schema, I am successfully able to query all entries from the Blog table. I am also able to query from the Post table ONLY IF a value for "blog" field exists for all entries inside the Post table on DynamoDB.

The issue (specific to iOS only) arises when:

1. There is no value for "blog" for a particular entry and that entry is queried or is part of the result from a query.
2. When I try to create a Post and deliberately leave out "blog" OR set "blog" to nil

I have also tried setting no value or nil for the "random_id" field but that does not cause any issues. Only "blog" field causes issues.

I see this in logs whenever it happens in case of both query and save operations:

Error creating post - DataStoreError: The data couldn’t be read because it is missing. Recovery suggestion: The data couldn’t be read because it is missing. Caused by: valueNotFound(Swift.String, Swift.DecodingError.Context(codingPath: [CodingKeys(stringValue: "elements", intValue: nil), _JSONKey(stringValue: "Index 0", intValue: 0), CodingKeys(stringValue: "blog", intValue: nil), CodingKeys(stringValue: "id", intValue: nil)], debugDescription: "Expected String value but found null instead.", underlyingError: nil))

There was an issue with the CLI when specifying "null" value for optional Index fields. However, that issue was fixed in CLI version 8.0.2 and works perfectly on Android.

Here are the queries that I am using:

Query for Blog and Save operation for Post

    func queryItem() {
        var blog: Blog? = nil
        Amplify.DataStore.query(Blog.self) {
            switch $0 {
            case .success(let result):
                print("Items: \(result)")
                blog = result[0]
                createItem(item: blog!)
            case .failure(let error):
                print("Error listing items - \(error)")
            }
        }
    }
    
    func createItem(item: Blog) {
        Amplify.DataStore.save(Post(name: "Test-2")) {
            switch $0 {
            case .success:
                print("Created a new post successfully")
            case .failure(let error):
                print("Error creating post - \(error)")
            }
        }
    }

Steps To Reproduce

Steps to reproduce the behavior:
1. Set up app backend with the provided schema
2. Populate data
4. Populate some entries inside `Post` model without specifying value for blog or set blog = nil
5. See error when creating , updating, or querying any entry with missing blog.

Expected behavior

I should be able to list or update all Posts irrespective of whether or not a value for "blog" exists.

Amplify Framework Version

1.24.0

Amplify Categories

DataStore

Dependency manager

Swift PM

Swift version

5.6

CLI version

8.2.0

Xcode version

13.3.1

Relevant log output

Error creating post - DataStoreError: The data couldn’t be read because it is missing.
Recovery suggestion: The data couldn’t be read because it is missing.
Caused by:
valueNotFound(Swift.String, Swift.DecodingError.Context(codingPath: [CodingKeys(stringValue: "elements", intValue: nil), _JSONKey(stringValue: "Index 0", intValue: 0), CodingKeys(stringValue: "blog", intValue: nil), CodingKeys(stringValue: "id", intValue: nil)], debugDescription: "Expected String value but found null instead.", underlyingError: nil))

Is this a regression?

No

Regression additional context

No response

Device

iPhone SE (3rd Generation)

iOS Version

15.4

Specific to simulators

No response

Additional context

No response

[srimalaya]

An update, in case it helps: I tried using GraphQL Mutations (with the same code-generated models). GQL mutations work perfectly even when "blog" is null. I am able to both query and create a post without specifying a blog and it works as expected.

So, there's probably nothing wrong with the models, but a bug on the DataStore end is causing this issue. This is a serious constraint on our end. On our actual app, the schema has several fields which are optional, which are then connected via a one-to-many relationship with another table. It will be a headache to accommodate this constraint which for some reason is specific only to iOS, because of which we didn't catch it earlier.

[lawmicha]

Hi @shri-onecup , i can confirm with the following steps that I am able to reproduce errors related to your schema.

1. On AppSync console, send a create mutation for a Post without a Blog. The post is created successfully.
2. In the sample app, pull the data down via Amplify.DataStore.start, no errors and the single post should be in the local store now.
3. Perform Amplify.DataStore.query(Post.self) and see the error

2022-05-19 10:05:50.739374-0400 amplify1792[63462:867886] [SQLiteStorageEngineAdapter] select
  "root"."id" as "id", "root"."createdAt" as "createdAt", "root"."name" as "name",
  "root"."random_id" as "random_id", "root"."updatedAt" as "updatedAt", "root"."blog_id" as "blog_id",
  "blog"."id" as "blog.id", "blog"."createdAt" as "blog.createdAt", "blog"."customs" as "blog.customs",
  "blog"."name" as "blog.name", "blog"."notes" as "blog.notes", "blog"."updatedAt" as "blog.updatedAt"
from "Post" as "root"
left outer join "Blog" as "blog"
  on "blog"."id" = "root"."blog_id"
2022-05-19 10:05:50.742229-0400 amplify1792[63462:867886] [SQLiteStorageEngineAdapter] select
  "root"."id" as "id", "root"."createdAt" as "createdAt", "root"."name" as "name",
  "root"."random_id" as "random_id", "root"."updatedAt" as "updatedAt", "root"."blog_id" as "blog_id",
  "blog"."id" as "blog.id", "blog"."createdAt" as "blog.createdAt", "blog"."customs" as "blog.customs",
  "blog"."name" as "blog.name", "blog"."notes" as "blog.notes", "blog"."updatedAt" as "blog.updatedAt"
from "Post" as "root"
left outer join "Blog" as "blog"
  on "blog"."id" = "root"."blog_id"
2022-05-19 10:05:50.742564-0400 amplify1792[63462:867886] [DataStore] A column named blog_id was found in the result set but no field on
Post could be found with that name and it will be ignored.
2022-05-19 10:05:50.742806-0400 amplify1792[63462:867886] [DataStore] A column named blog_id was found in the result set but no field on
Post could be found with that name and it will be ignored.
DataStoreError: The data couldn’t be read because it is missing.
Recovery suggestion: The data couldn’t be read because it is missing.
Caused by:
valueNotFound(Swift.String, Swift.DecodingError.Context(codingPath: [CodingKeys(stringValue: "elements", intValue: nil), _JSONKey(stringValue: "Index 0", intValue: 0), CodingKeys(stringValue: "blog", intValue: nil), CodingKeys(stringValue: "id", intValue: nil)], debugDescription: "Expected String value but found null instead.", underlyingError: nil))

I believe this is related to queries performing an eager load on the associated model (the Blog of the Post). I believe the expected behavior should be to return the Post successful with nil blog. We'll have to look into this more closely in where exactly it's failing, how the SQL statement is created, and figure out a fix that decodes the data properly

[srimalaya]

@lawmicha Thanks for the update! Since, this is actually a bug with DataStore, can you please give me an estimate on when I can expect a fix. This bug is causing serious setbacks on our iOS app's production pipeline because all our other platforms (JS and Android) rely solely on DataStore.

[srimalaya]

@lawmicha can you please provide an update? We are really stalled on our timeline to launch the app into production. It is especially troublesome because I am forced to use GraphQL for now while both our other platforms are using DataStore, because of which we are facing issues with offline use, and updates to existing entries.

[lawmicha]

Hi @shri-onecup, sorry for the delays, the branch for this fix is located at fix/optional-associations please give it a try and let us know if you are seeing any issues. The PR is currently in draft as I work through more testing: https://github.com/aws-amplify/amplify-ios/pull/1849

The following use cases i've tested:

• save a post and query for it
• save a blog and query for it
• save a post, save a blog, update the post, query the post and see that the blog is eager loaded, query the blog and lazy load the posts.
• query all post, can see that some posts belong to a blog and some do not

[srimalaya]

Hi @lawmicha, I just got a chance to do some quick tests and it seems like the fix is working perfectly on my end. Thanks for it. Any ideas on when it might be released officially?

[srimalaya]

Hi @lawmicha, any estimates on when the fix will be made official? I am really looking forward to using it as part of an official release ASAP.

[srimalaya]

@lawmicha I think ObserveQuery is still facing this problem. The initial sync runs fine. But when fetching a real-time change from cloud, I get this error and the subscriptions just stop and don't start again even when cancelled and subscribed to again via another observeQuery. [IncomingAsyncSubscriptionEventToAnyModelMapper] The operation couldn’t be completed. (Amplify.GraphQLResponseError<AWSPluginsCore.MutationSync<AWSPluginsCore.AnyModel>> error 1.)

Here's a bit detailed log (I cut chunks off it. Let me know if you want the entirety):

[IncomingAsyncSubscriptionEventPublisher] onUpdateValueListener: data(Swift.Result<AWSPluginsCore.MutationSync<AWSPluginsCore.AnyModel>, Amplify.GraphQLResponseError<AWSPluginsCore.MutationSync<AWSPluginsCore.AnyModel>>>.failure(GraphQLResponseError<MutationSync<AnyModel>>: GraphQL service returned a partially-successful response containing errors: [Amplify.GraphQLError(message: "Cannot return null for non-nullable type: \'Sex\' within parent \'Animal\' (/onUpdateNote/animal/animal_sex)", locations: nil, path: Optional([Amplify.JSONValue.string("onUpdateNote"), Amplify.JSONValue.string("animal"), Amplify.JSONValue.string("animal_sex")]), extensions: nil), .........same message for other fields......]

[IncomingAsyncSubscriptionEventToAnyModelMapper] dispose(of graphQLResponse): failure(GraphQLResponseError<MutationSync<AnyModel>>: GraphQL service returned a partially-successful response containing errors: [Amplify.GraphQLError(message: "Cannot return null for non-nullable type: \'Sex\' within parent \'Animal\' (/onUpdateNote/animal/animal_sex)", locations: nil, path: Optional([Amplify.JSONValue.string("onUpdateNote"), Amplify.JSONValue.string("animal"), Amplify.JSONValue.string("animal_sex")]), extensions: nil), .........same message for other fields......]

Recovery suggestion: The list of `GraphQLError` contains service-specific messages.)
2022-06-17 09:11:18.026604-0700 user-app[16800:26825656] [IncomingAsyncSubscriptionEventToAnyModelMapper] The operation couldn’t be completed. (Amplify.GraphQLResponseError<AWSPluginsCore.MutationSync<AWSPluginsCore.AnyModel>> error 1.)

For your reference: I was observing a Note model which has an optional field animal linked with @belongsTo directive.

[lawmicha]

Hi @shri-onecup, thanks for taking the time to test and confirm that it works for you. I see the PR with the fix merged but not released yet. The current latest is 1.26.2 so it should go out in 1.26.3 (or 1.27.0). Will update here once there is a release. If you like, i believe there's an unstable release you can use if you want to move off the branch and additional changes from main. The unstable release version is up to 1.26.3-unstable.9 now.

[lawmicha]

For the issue, @shri-onecup could you please provide the schema (redacted if needed) containing the Note, Animal, and related models? ObserveQuery is an aggregation API that leverages the real-time APIs underneath the covers. The sync process in DataStore establishes the AppSync subscriptions, and events that reconcile to the local store are emitted as real time events for the ObserveQuery to generate snapshots on.

That is to say, we should be able to reproduce this issue by using the schema that you provide, starting the sync process with DataStore.start(), saving a Note model, then updating it to seeing the error on the onUpdateNote subscription.

Some scenarios we should test is

• Saving a Note with null belongs-to relationship
• Updating a Note with null belongs-to relationship
• Saving the parent model, then saving the Note with belongs-to parent
• Updating the Note that has belongs-to parent by setting to nil on the belongs to.

Each save and update should have corresponding subscription events successfully reconciled

[lawmicha]

Also, are you able to confirm that the subscription error on iOS occurs during a local iOS DataStore.save or is it from a different app, say Android? I've seen issues where other platforms produce a mutation (from DataStore.save) with a selection set that does not match the expected selection set on iOS DataStore's subscription operation. This misalignment in selection set is what causes the events to be received by iOS but missing some data for proper decoding

[srimalaya]

Hi @lawmicha I am still using the #1849 branch to make my app work at the moment. Let me know if you discourage that.

I see this when making an update on our web (JavaScript) Client only.

2022-07-12 12:03:40.556933-0700 user-app[98581:1674838] [IncomingAsyncSubscriptionEventToAnyModelMapper] The operation couldn’t be completed. (Amplify.GraphQLResponseError<AWSPluginsCore.MutationSync<AWSPluginsCore.AnyModel>> error 1.)

Interestingly enough, when I run the observeQuery again after this, I get the isSynced=true flag but the data that I receive still shows old data. However, if I restart the app, I can see the change there. This would mean that the update is indeed pulled in but it is never pushed within a query for some reason.

I also tried doing an update on AppSync and when doing that, the ObserveQuery does not give any logs. Neither any incoming updates nor any errors like in case of the web client. I only see new data when I deliberately either do Amplify.clear() -> Amplify.start() or quit & restart the app.

I am running the tests that you suggest now on a new environment using a similar schema. I'll update you as soon as possible for this. I cannot possibly share the schema publicly because of how complex it is and I cannot safely redact information. For that reason, I will attempt to create a schema with similar connections and see if I can reproduce this behaviour.

[srimalaya]

@lawmicha I switched to 1.27.0 yesterday. It seemed to work fine but today I tried to clear the local database and do DataStore.Start() and I see the following errors for every single model that has optionals (almost all our models):

Caused by:
GraphQLResponseError<PaginatedList<AnyModel>>: GraphQL service returned a successful response containing errors: [Amplify.GraphQLError(message: "Encountered \"#end\\n\" at velocity[line 18, column 1]\nWas expecting one of:\n    <EOF> \n    \"(\" ...\n    <RPAREN> ...\n    <ESCAPE_DIRECTIVE> ...\n    <SET_DIRECTIVE> ...\n    \"##\" ...\n    \"\\\\\\\\\" ...\n    \"\\\\\" ...\n    <TEXT> ...\n    \"*#\" ...\n    \"*#\" ...\n    \"]]#\" ...\n    <STRING_LITERAL> ...\n    <IF_DIRECTIVE> ...\n    <INTEGER_LITERAL> ...\n    <FLOATING_POINT_LITERAL> ...\n    <WORD> ...\n    <BRACKETED_WORD> ...\n    <IDENTIFIER> ...\n    <DOT> ...\n    \"{\" ...\n    \"}\" ...\n    <EMPTY_INDEX> ...\n    ", locations: Optional([Amplify.GraphQLError.Location(line: 2, column: 3)]), path: Optional([Amplify.JSONValue.string("syncMeasurements")]), extensions: Optional(["data": Amplify.JSONValue.null, "errorType": Amplify.JSONValue.string("MappingTemplate"), "errorInfo": Amplify.JSONValue.null]))]
Recovery suggestion: The list of `GraphQLError` contains service-specific messages
DataStoreError: An error occurred syncing Treatment
Caused by:
DataStoreError: GraphQL service returned a successful response containing errors: [Amplify.GraphQLError(message: "Encountered \"#end\\n\" at velocity[line 18, column 1]\nWas expecting one of:\n    <EOF> \n    \"(\" ...\n    <RPAREN> ...\n    <ESCAPE_DIRECTIVE> ...\n    <SET_DIRECTIVE> ...\n    \"##\" ...\n    \"\\\\\\\\\" ...\n    \"\\\\\" ...\n    <TEXT> ...\n    \"*#\" ...\n    \"*#\" ...\n    \"]]#\" ...\n    <STRING_LITERAL> ...\n    <IF_DIRECTIVE> ...\n    <INTEGER_LITERAL> ...\n    <FLOATING_POINT_LITERAL> ...\n    <WORD> ...\n    <BRACKETED_WORD> ...\n    <IDENTIFIER> ...\n    <DOT> ...\n    \"{\" ...\n    \"}\" ...\n    <EMPTY_INDEX> ...\n    ", locations: Optional([Amplify.GraphQLError.Location(line: 2, column: 3)]), path: Optional([Amplify.JSONValue.string("syncTreatments")]), extensions: Optional(["errorType": Amplify.JSONValue.string("MappingTemplate"), "data": Amplify.JSONValue.null, "errorInfo": Amplify.JSONValue.null]))]
Recovery suggestion: The list of `GraphQLError` contains service-specific messages
Caused by:
GraphQLResponseError<PaginatedList<AnyModel>>: GraphQL service returned a successful response containing errors: [Amplify.GraphQLError(message: "Encountered \"#end\\n\" at velocity[line 18, column 1]\nWas expecting one of:\n    <EOF> \n    \"(\" ...\n    <RPAREN> ...\n    <ESCAPE_DIRECTIVE> ...\n    <SET_DIRECTIVE> ...\n    \"##\" ...\n    \"\\\\\\\\\" ...\n    \"\\\\\" ...\n    <TEXT> ...\n    \"*#\" ...\n    \"*#\" ...\n    \"]]#\" ...\n    <STRING_LITERAL> ...\n    <IF_DIRECTIVE> ...\n    <INTEGER_LITERAL> ...\n    <FLOATING_POINT_LITERAL> ...\n    <WORD> ...\n    <BRACKETED_WORD> ...\n    <IDENTIFIER> ...\n    <DOT> ...\n    \"{\" ...\n    \"}\" ...\n    <EMPTY_INDEX> ...\n    ", locations: Optional([Amplify.GraphQLError.Location(line: 2, column: 3)]), path: Optional([Amplify.JSONValue.string("syncTreatments")]), extensions: Optional(["errorType": Amplify.JSONValue.string("MappingTemplate"), "data": Amplify.JSONValue.null, "errorInfo": Amplify.JSONValue.null]))]
Recovery suggestion: The list of `GraphQLError` contains service-specific messages
DataStoreError: An error occurred syncing Wifi
Caused by:
DataStoreError: GraphQL service returned a successful response containing errors: [Amplify.GraphQLError(message: "Encountered \"#end\\n\" at velocity[line 18, column 1]\nWas expecting one of:\n    <EOF> \n    \"(\" ...\n    <RPAREN> ...\n    <ESCAPE_DIRECTIVE> ...\n    <SET_DIRECTIVE> ...\n    \"##\" ...\n    \"\\\\\\\\\" ...\n    \"\\\\\" ...\n    <TEXT> ...\n    \"*#\" ...\n    \"*#\" ...\n    \"]]#\" ...\n    <STRING_LITERAL> ...\n    <IF_DIRECTIVE> ...\n    <INTEGER_LITERAL> ...\n    <FLOATING_POINT_LITERAL> ...\n    <WORD> ...\n    <BRACKETED_WORD> ...\n    <IDENTIFIER> ...\n    <DOT> ...\n    \"{\" ...\n    \"}\" ...\n    <EMPTY_INDEX> ...\n    ", locations: Optional([Amplify.GraphQLError.Location(line: 2, column: 3)]), path: Optional([Amplify.JSONValue.string("syncWifis")]), extensions: Optional(["errorType": Amplify.JSONValue.string("MappingTemplate"), "data": Amplify.JSONValue.null, "errorInfo": Amplify.JSONValue.null]))]
Recovery suggestion: The list of `GraphQLError` contains service-specific messages
Caused by:
GraphQLResponseError<PaginatedList<AnyModel>>: GraphQL service returned a successful response containing errors: [Amplify.GraphQLError(message: "Encountered \"#end\\n\" at velocity[line 18, column 1]\nWas expecting one of:\n    <EOF> \n    \"(\" ...\n    <RPAREN> ...\n    <ESCAPE_DIRECTIVE> ...\n    <SET_DIRECTIVE> ...\n    \"##\" ...\n    \"\\\\\\\\\" ...\n    \"\\\\\" ...\n    <TEXT> ...\n    \"*#\" ...\n    \"*#\" ...\n    \"]]#\" ...\n    <STRING_LITERAL> ...\n    <IF_DIRECTIVE> ...\n    <INTEGER_LITERAL> ...\n    <FLOATING_POINT_LITERAL> ...\n    <WORD> ...\n    <BRACKETED_WORD> ...\n    <IDENTIFIER> ...\n    <DOT> ...\n    \"{\" ...\n    \"}\" ...\n    <EMPTY_INDEX> ...\n    ", locations: Optional([Amplify.GraphQLError.Location(line: 2, column: 3)]), path: Optional([Amplify.JSONValue.string("syncWifis")]), extensions: Optional(["errorType": Amplify.JSONValue.string("MappingTemplate"), "data": Amplify.JSONValue.null, "errorInfo": Amplify.JSONValue.null]))]
Recovery suggestion: The list of `GraphQLError` contains service-specific messages
DataStoreError: An error occurred syncing Note
Caused by:
DataStoreError: GraphQL service returned a successful response containing errors: [Amplify.GraphQLError(message: "Encountered \"#end\\n\" at velocity[line 18, column 1]\nWas expecting one of:\n    <EOF> \n    \"(\" ...\n    <RPAREN> ...\n    <ESCAPE_DIRECTIVE> ...\n    <SET_DIRECTIVE> ...\n    \"##\" ...\n    \"\\\\\\\\\" ...\n    \"\\\\\" ...\n    <TEXT> ...\n    \"*#\" ...\n    \"*#\" ...\n    \"]]#\" ...\n    <STRING_LITERAL> ...\n    <IF_DIRECTIVE> ...\n    <INTEGER_LITERAL> ...\n    <FLOATING_POINT_LITERAL> ...\n    <WORD> ...\n    <BRACKETED_WORD> ...\n    <IDENTIFIER> ...\n    <DOT> ...\n    \"{\" ...\n    \"}\" ...\n    <EMPTY_INDEX> ...\n    ", locations: Optional([Amplify.GraphQLError.Location(line: 2, column: 3)]), path: Optional([Amplify.JSONValue.string("syncNotes")]), extensions: Optional(["data": Amplify.JSONValue.null, "errorType": Amplify.JSONValue.string("MappingTemplate"), "errorInfo": Amplify.JSONValue.null]))]
Recovery suggestion: The list of `GraphQLError` contains service-specific messages
Caused by:
GraphQLResponseError<PaginatedList<AnyModel>>: GraphQL service returned a successful response containing errors: [Amplify.GraphQLError(message: "Encountered \"#end\\n\" at velocity[line 18, column 1]\nWas expecting one of:\n    <EOF> \n    \"(\" ...\n    <RPAREN> ...\n    <ESCAPE_DIRECTIVE> ...\n    <SET_DIRECTIVE> ...\n    \"##\" ...\n    \"\\\\\\\\\" ...\n    \"\\\\\" ...\n    <TEXT> ...\n    \"*#\" ...\n    \"*#\" ...\n    \"]]#\" ...\n    <STRING_LITERAL> ...\n    <IF_DIRECTIVE> ...\n    <INTEGER_LITERAL> ...\n    <FLOATING_POINT_LITERAL> ...\n    <WORD> ...\n    <BRACKETED_WORD> ...\n    <IDENTIFIER> ...\n    <DOT> ...\n    \"{\" ...\n    \"}\" ...\n    <EMPTY_INDEX> ...\n    ", locations: Optional([Amplify.GraphQLError.Location(line: 2, column: 3)]), path: Optional([Amplify.JSONValue.string("syncNotes")]), extensions: Optional(["data": Amplify.JSONValue.null, "errorType": Amplify.JSONValue.string("MappingTemplate"), "errorInfo": Amplify.JSONValue.null]))]
Recovery suggestion: The list of `GraphQLError` contains service-specific messages

[lawmicha]

Hi @shri-onecup,

I see this when making an update on our web (JavaScript) Client only

2022-07-12 12:03:40.556933-0700 user-app[98581:1674838] [IncomingAsyncSubscriptionEventToAnyModelMapper] The operation couldn’t be completed. (Amplify.GraphQLResponseError<AWSPluginsCore.MutationSync<AWSPluginsCore.AnyModel>> error 1.)

I believe what you're seeing here is most likely due to a selection set mismatch across the DataStore clients on different platforms. We have an issue open with some more details in here https://github.com/aws-amplify/amplify-ios/issues/1753

If the selection set mismatch exists (iOS expects more data to be returned on the subscription than what was sent from the mutation on the JS client) then the iOS client will fail to decode the data. You'll see that restarting the app, DataStore will perform the sync process which does a syncQuery operation to pull the data in successfully.

The update on AppSync needs to be carefully crafted to ensure the selection set in the update is what iOS clients expect as well.

[lawmicha]

For this error,

GraphQLResponseError<PaginatedList<AnyModel>>: GraphQL service returned a successful response containing errors: [Amplify.GraphQLError(message: "Encountered \"#end\\n\" at velocity[line 18, column 1]\nWas expecting one of:\n    <EOF> \n    \"(\" ...\n    <RPAREN> ...\n    <ESCAPE_DIRECTIVE> ...\n    <SET_DIRECTIVE> ...\n    \"##\" ...\n    \"\\\\\\\\\" ...\n    \"\\\\\" ...\n    <TEXT> ...\n    \"*#\" ...\n    \"*#\" ...\n    \"]]#\" ...\n    <STRING_LITERAL> ...\n    <IF_DIRECTIVE> ...\n    <INTEGER_LITERAL> ...\n    <FLOATING_POINT_LITERAL> ...\n    <WORD> ...\n    <BRACKETED_WORD> ...\n    <IDENTIFIER> ...\n    <DOT> ...\n    \"{\" ...\n    \"}\" ...\n    <EMPTY_INDEX> ...\n    ", locations: Optional([Amplify.GraphQLError.Location(line: 2, column: 3)]), path: Optional([Amplify.JSONValue.string("syncMeasurements")]), extensions: Optional(["data": Amplify.JSONValue.null, "errorType": Amplify.JSONValue.string("MappingTemplate"), "errorInfo": Amplify.JSONValue.null]))]

can you try running the syncQuery operation in the AppSync console and see if you can get the response in a clearer format, with the focus on Encountered \"#end\\n\" at velocity[line 18, column 1]\nWas expecting one of: ?

Looks like there are some changes on the server side or data in the remote DB that is causing some problems when it is being queried

[srimalaya]

@lawmicha thanks for the reply. I will go through the JS code now. But how do I find what fields are missing that iOS is expecting? I tried using CloudWatch to see if I can figure the missing fields out but I can't see what gets pushed to iOS when an onUpdate mutation is received by AppSync from the Javascript client. I can see what JS pushed to AppSync but that is not enough to figure it out.

As for the other issue, I don't get the error when using AppSync console. I'll try to get a clearer message.

[lawmicha]

if the JS Client is using JS DataStore, then internally the sync process is what's generating the selection set when performing the mutation to AppSync. This isn't something you can control and is pending a JS fix to populate the remaining fields required for iOS's DataStore's sync process to properly receive subscription events. The team is aware of this and investigating what changes we need to do.

If the JS Client is making a direct API call to AppSync, from an AWS Lambda for example, and the selection set is controllable, then it should be populated with all the fields, and nested belongs-to fields, and system metadata fields (__typename, _version, _deleted, _lastChangedAt).

For example, on iOS, this is the selection created for a Post

createPost(input: $input) {
  id
  createdAt
  rating
  title
  updatedAt
 blog {
    id
    createdAt
    name
    updatedAt
    __typename
    _version
    _deleted
    _lastChangedAt
  }
  __typename
  _version
  _deleted
  _lastChangedAt
}

On JS, this is what's created for the same mutation through DataStore, which is generating subscription events that cannot be processed by the iOS DataStore clients.

createPost(input: $input) {
  id
  title
  rating
  createdAt
  updatedAt
  _version
  _lastChangedAt
  _deleted
 blog {
    id
    _deleted
  }
}

[srimalaya]

@lawmicha I am pretty sure this is an error on iOS only and it is a bug on iOS (or the underlying connection between AppSync and Amplify-iOS’ rather than JavaScript or Android. I just tested this on Android as well. I see the exact same error when I make changes on Android and the subscription that is running on iOS attempts to pull them.

I think it is happening on iOS now because AppSync does not push values for nested models to iOS when pushing them via subscriptions. For example, for this schema:

type Blog @model @auth(rules: [{allow: public}]) {
  id: ID!
  name: String!
  customs: [MyCustomModel]
  notes: [String]
  post: [Post] @hasMany(indexName: "postByBlog", fields: ["id"])
}

type Post @model @auth(rules: [{allow: public}]) {
  id: ID!
  name: String!
  blog_id: ID @index(name: "postByBlog")
  random_id: String @index(name: "byRandom")
  blog: Blog @belongsTo(fields: ["blog_id"])
}

type MyCustomModel {
  id: ID!
  name: String!
  desc: String
  children: [MyNestedModel]
}

type MyNestedModel {
  id: ID!
  nestedName: String!
  notes: [String]
}

On Android or JavaScript, do the following:

On iOS (Swift)
1. Do `Amplify.DataStore.Start()` upon app launch, then start a subscription (or `ObserveQuery`) for `Post` model.

On Android/JavaScript
1. Create a `Post` (with just `name` as input).
2. Attempt to update that post. The update is promptly received by the running subscription on iOS
3. Create a `Blog` object with just `name` as input. Note the newly created blog's ID.
4. Create another new `Post` but this time input `name`, & `blog`. Use the `blog` created in "step 4" for this step.
5. Notice that the subscription running on the `Post` table promptly receives this update as well.
6. Now, try to update just the `name` field in the post object created in step 5. 
7. See the error.

While doing the above steps, I was Logging every single step. The object that was pushed using the DataStore.save() operation did in-fact have values for blog. I also confirmed it on CloudWatch that the save operation did indeed push all available values, including those for blog. But on iOS, using Logging level of Verbose, I see the following error when the subscription is attempting to pull the newly changed value in Step 6:

[Amplify.GraphQLError(message: "Cannot return null for non-nullable type: \'AWSDateTime\' within parent \'Blog\' (/onUpdatePost/blog/createdAt)", locations: nil, path: Optional([Amplify.JSONValue.string("onUpdatePost"), Amplify.JSONValue.string("blog")

I tried GraphQL subscription as well and even that throws the exact same error. This does not happen for subscriptions running on Android or JavaScript and is only affecting iOS.

[srimalaya]

Hi @lawmicha , can you link the appropriate ticket that targets the problem with subscriptions or give me an update on the progress if you're aware of it? This issue of subscriptions (on both GraphQL & DataStore) not working cross-platform between iOS and JS/Android is really holding us back from deploying a synchronized system to our clients.

I would also appreciate it if you could also push this issue up internally with the JS (or CLI) team as I am sure we are not the only ones suffering from this.

I can open a new issue as well if you'd like me to.

[chrisbonifacio]

@shri-onecup Apologies for the delay, we've decided on the direction we want to take on addressing this issue and similar ones with selection sets and making them compatible across platforms. Please know that we are working on this internally and will reply back with updates as they come.

We are using this issue to track our progress: https://github.com/aws-amplify/amplify-ios/issues/1753

[lawmicha]

TLDR: Just an update on this, the first issue related to optional associations has been fixed on the iOS client side (https://github.com/aws-amplify/amplify-swift/pull/1849 for more details). The latest issue is regarding JS mutation events not being reconcilied by the iOS client successfully. This is fixed, pending the Lazy Loading and Custom Selection Set capabilities introduced in the PR https://github.com/aws-amplify/amplify-swift/pull/2583

Long version:

The subscription request's selection set defined by the iOS client requires more info than the mutation request's selection set from the JS client. There are two directions to make these two platforms compatible, either JS increases their selection set to provide all the data required by the iOS client, or iOS client should be able to operate against the smaller response payload sourced from the JS client. Our approach is the latter, to make the iOS client able to operate successfully against the response payload sourced from JS clients.

The PR (https://github.com/aws-amplify/amplify-swift/pull/2583) and related codegen PRs will be required to successfully verify that the problem is fixed for this use case. The changes in the PR introduces a wrapper on the belongs-to association as a LazyReference type that can be used to decode response payloads that only contain the primary keys. The second relevant change here is that the subscription selection set created by the iOS client has been reduced, to only containing the selection set of the primary keys for the nested model. The third subtle change is that iOS clients still have a dependency on the __typename to be at the first level of the response data and uses that to decode to the type, the API Plugin (used by DataStore) will check when the __typename does not exist, injects it into the response payload before continuing decoding. There's more than one way to accomplish this, but this is the least lift needed- that is we're not removing its dependency on __typename but working around it for JS clients that send mutations without __typename in the selection set. In the long term, we want to remove the __typename from all selection sets to improve the latency of the request.

To verify this, we should create the JS app using the schema provided. We can also do this from Studio since Studio uses the JS library to send mutations. The schema is

type Blog @model @auth(rules: [{allow: public}]) {
  id: ID!
  name: String!
  customs: [MyCustomModel]
  notes: [String]
  post: [Post] @hasMany(indexName: "postByBlog", fields: ["id"])
}

type Post @model @auth(rules: [{allow: public}]) {
  id: ID!
  name: String!
  blog_id: ID @index(name: "postByBlog")
  random_id: String @index(name: "byRandom")
  blog: Blog @belongsTo(fields: ["blog_id"])
}

type MyCustomModel {
  id: ID!
  name: String!
  desc: String
  children: [MyNestedModel]
}

type MyNestedModel {
  id: ID!
  nestedName: String!
  notes: [String]
}

Notes for the team, our current steps to repro (will be simplified once the PR's get merged and released)

1. Create the backend, with the client application as a JS app or use Studio's data creation tab
2. Create the iOS app, that depends on the library PR (https://github.com/aws-amplify/amplify-swift/pull/2583) (use the branch as the SPM branch target)
3. Codegen the new types using https://github.com/aws-amplify/amplify-codegen/pull/504 (see PR description for local set up)
4. Start the iOS app, and call DataStore.start(), send the mutation sourced from JS library, and expect the data to be successfully reconciled.

[royjit]

Amplify Swift changes are released, waiting for Amplify Codegen changes to be released to do a final round of testing.

[royjit]

Tested and verified in the new amplify-swift version 2.6.1 and models generated by amplify-cli version 11.0.3. Before generating models make sure to enable generatemodelsforlazyloadandcustomselectionset to true inside cli.json file.

Here is how my cli.json looks like:

{
  "features": {
    "graphqltransformer": {
      "addmissingownerfields": true,
      "improvepluralization": false,
      "validatetypenamereservedwords": true,
      "useexperimentalpipelinedtransformer": true,
      "enableiterativegsiupdates": true,
      "secondarykeyasgsi": true,
      "skipoverridemutationinputtypes": true,
      "transformerversion": 2,
      "suppressschemamigrationprompt": true,
      "securityenhancementnotification": false,
      "showfieldauthnotification": false,
      "usesubusernamefordefaultidentityclaim": true,
      "usefieldnameforprimarykeyconnectionfield": false,
      "enableautoindexquerynames": true,
      "respectprimarykeyattributesonconnectionfield": true,
      "shoulddeepmergedirectiveconfigdefaults": false,
      "populateownerfieldforstaticgroupauth": true
    },
    "frontend-ios": {
      "enablexcodeintegration": true
    },
    "auth": {
      "enablecaseinsensitivity": true,
      "useinclusiveterminology": true,
      "breakcirculardependency": true,
      "forcealiasattributes": false,
      "useenabledmfas": true
    },
    "codegen": {
      "useappsyncmodelgenplugin": true,
      "usedocsgeneratorplugin": true,
      "usetypesgeneratorplugin": true,
      "cleangeneratedmodelsdirectory": true,
      "retaincasestyle": true,
      "addtimestampfields": true,
      "handlelistnullabilitytransparently": true,
      "emitauthprovider": true,
      "generateindexrules": true,
      "enabledartnullsafety": true,
      "generatemodelsforlazyloadandcustomselectionset": true
    },
    "appsync": {
      "generategraphqlpermissions": true
    },
    "latestregionsupport": {
      "pinpoint": 1,
      "translate": 1,
      "transcribe": 1,
      "rekognition": 1,
      "textract": 1,
      "comprehend": 1
    },
    "project": {
      "overrides": true
    }
  },
  "debug": {
    "shareProjectConfig": false
  }
}