Re-arrange E2E tests

[KeranYang]

As we introduce more features and SDKs, the number of e2e test cases grows. It's important to follow certain standards to ensure we are writing the right test cases and NOT introduce unnecessary ones. It's observed that having a high number of test cases within one test suite can cause flakiness in CI workflow.

The goal of this task is to see if we can reduce the number of e2e test cases and still maintain the coverage, hence resolving the flakiness issue.

There are two types of e2e tests, one is to test SDKs and the other platform features. Without sacrificing coverage, I am proposing the following guidelines for adding/re-arranging e2e tests:

1. SDK tests - There should be only one test case for each (language, SDK) combination. SDK tests are acceptance tests where we just verify that the platform can successfully communicate with the SDK. More complicated tests like mocking the input stream and verifying the output stream should be covered in the SDK repository using unit tests.
2. Platform tests - If a platform test case A is also covered by another case, we can remove A.
3. It's recommended that the platform tests all use Golang SDKs, which are more stable.
4. If a test suite grows and takes close to 15 minutes to finish, divide it into two.

Message from the maintainers:

Impacted by this bug? Give it a 👍. We often sort issues this way to know what to prioritize.

For quick help and support, join our slack channel.

[KeranYang]

@yhl25 suggested and I agreed that for each language, we can have a single pipeline that covers all interfaces, sourcer, source transformer, mapper, etc. That way whenever we update the SDK, by simply testing this one pipeline, we verify the accuracy of our changes.