DataStore V1 Integration Tests Stability

[lawmicha]

Verification when a new PR is created

Integration Tests to verify

1. DataStoreIntegrationTests (includes V2 and CPK)
2. DataStoreAuthTests (includes MultiAuth, Cognito Auth, IAM Auth)

Existing Failing tests

DataStoreAuthIntegrationTests and DataStoreIntegrationTests are mostly passing except for

• ~~DataStoreConnectionOptionalAssociations.testRemovePostFromCommentAndBlogFromPost~~ #2356
• ~~DataStoreConnectionScenario6Tests.testDeleteAll~~ #2358

Verified that they are still failing without the changes from this PR. We should figure out why these these are not passing anymore, but in parallel continue getting this critical bug fix into main then cherry-picked over to dev-preview.

Need to test

Flutter Tests

Originally posted by @lawmicha in https://github.com/aws-amplify/amplify-ios/issues/2319#issuecomment-1255193235

[github-actions[bot]]

This issue was opened by a maintainer of this repository; updates will be posted here. If you are also experiencing this issue, please comment here with any relevant information so that we're aware and can prioritize accordingly.

[lawmicha]

api-functional-tests are failing

AWSAPIPluginFunctionalTests.GraphQLConnectionScenario3Tests
  testListPostWithPredicate, XCTAssertEqual failed: ("0") is not equal to ("1")
  /Users/runner/work/amplify-swift/amplify-swift/AmplifyPlugins/API/Tests/APIHostApp/AWSAPIPluginFunctionalTests/GraphQLConnectionScenario3Tests+List.swift:138
  ``
        }
        XCTAssertEqual(posts.count, 1)
        guard let singlePost = posts.first else {
  ``

  testListPostWithPredicate, failed - Should only have a single post with the unique title
  /Users/runner/work/amplify-swift/amplify-swift/AmplifyPlugins/API/Tests/APIHostApp/AWSAPIPluginFunctionalTests/GraphQLConnectionScenario3Tests+List.swift:140
  ``
        guard let singlePost = posts.first else {
            XCTFail("Should only have a single post with the unique title")
            return
  ``

AWSAPIPluginFunctionalTests.GraphQLModelBasedTests
  testListQueryWithPredicate, XCTAssertEqual failed: ("0") is not equal to ("1")
  /Users/runner/work/amplify-swift/amplify-swift/AmplifyPlugins/API/Tests/APIHostApp/AWSAPIPluginFunctionalTests/GraphQLModelBasedTests.swift:186
  ``
        }
        XCTAssertEqual(posts.count, 1)
        guard let singlePost = posts.first else {
  ``

  testListQueryWithPredicate, failed - Should only have a single post with the unique title
  /Users/runner/work/amplify-swift/amplify-swift/AmplifyPlugins/API/Tests/APIHostApp/AWSAPIPluginFunctionalTests/GraphQLModelBasedTests.swift:188
  ``
        guard let singlePost = posts.first else {
            XCTFail("Should only have a single post with the unique title")
            return

Under GraphQLModelBasedTests

    ✗ testListQueryWithPredicate, XCTAssertEqual failed: ("0") is not equal to ("1")
    ✗ testListQueryWithPredicate, failed - Should only have a single post with the unique title

Running them against another backend and locally appears to be successful

In both tests, a model is created using the create mutation and then queried with a very specific predicate that should only match the one model that was created. It appears 50% of the test runs have passing tests for this test suite. The next step here is to figuer out why it's flaky

[lawmicha]

https://github.com/aws-amplify/amplify-swift/pull/2517#issuecomment-1303763916

Let's at a minimum update the README.md to instructions that make it clearer to provision the backend

for the multi auth tests involving OIDC

[royjit]

Adding a related issue here - https://github.com/aws-amplify/amplify-swift/issues/1882

[lawmicha]

triaged internally