[BUG] Flaky Tests on Data Prepper Trace Analytics Raw Span End-to-end test with Gradle

[laneholloway]

Describe the bug This integration test is failing randomly.

To Reproduce Submit a PR

Expected behavior The integration test should be deterministic in its passing or failing.

Environment (please complete the following information):

• OS: [e.g. Ubuntu 20.04 LTS]
• Version [e.g. 22]

Additional context It runs as part of our integration test suite. Here is a run that failed: https://github.com/opensearch-project/data-prepper/actions/runs/1386037215 the only thing that changed was a readme file.

[chenqi0805]

This also applies to service map end-to-end test. It is due to opensearch docker container shutdown unexpectedly. Here is the container logs:

...
[2022-03-18T21:37:56,571][INFO ][o.o.c.m.MetadataMappingService] [9b1afcc5a7ba] [security-auditlog-2022.03.18/78nQt25UTvSaY3Tpcbvnqw] update_mapping [_doc]
[2022-03-18T21:37:56,587][DEPRECATION][o.o.d.c.m.IndexNameExpressionResolver] [9b1afcc5a7ba] this request accesses system indices: [.opendistro_security], but in a future major version, direct access to system indices will be prevented by default
[2022-03-18T21:37:56,635][DEPRECATION][o.o.d.c.m.IndexNameExpressionResolver] [9b1afcc5a7ba] this request accesses system indices: [.opendistro_security], but in a future major version, direct access to system indices will be prevented by default
[2022-03-18T21:37:56,719][DEPRECATION][o.o.d.c.s.Settings       ] [9b1afcc5a7ba] [index.opendistro.index_state_management.rollover_alias] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2022-03-18T21:37:56,744][INFO ][o.o.c.m.MetadataCreateIndexService] [9b1afcc5a7ba] [otel-v1-apm-service-map] creating index, cause [api], templates [otel-v1-apm-service-map-index-template], shards [1]/[1]
[2022-03-18T21:37:56,856][DEPRECATION][o.o.d.c.s.Settings       ] [9b1afcc5a7ba] [index.opendistro.index_state_management.rollover_alias] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2022-03-18T21:37:56,969][DEPRECATION][o.o.d.c.m.IndexNameExpressionResolver] [9b1afcc5a7ba] this request accesses system indices: [.opendistro_security], but in a future major version, direct access to system indices will be prevented by default
[2022-03-18T21:37:57,062][INFO ][o.o.c.m.MetadataMappingService] [9b1afcc5a7ba] [security-auditlog-2022.03.18/78nQt25UTvSaY3Tpcbvnqw] update_mapping [_doc]
[2022-03-18T21:37:57,319][INFO ][o.o.c.m.MetadataMappingService] [9b1afcc5a7ba] [security-auditlog-2022.03.18/78nQt25UTvSaY3Tpcbvnqw] update_mapping [_doc]
Killing opensearch process 104
[2022-03-18T21:38:01,067][INFO ][o.o.s.a.r.AuditMessageRouter] [9b1afcc5a7ba] Closing AuditMessageRouter
[2022-03-18T21:38:01,069][INFO ][o.o.n.Node               ] [9b1afcc5a7ba] stopping ...
[2022-03-18T21:38:01,071][INFO ][o.o.s.a.s.SinkProvider   ] [9b1afcc5a7ba] Closing InternalOpenSearchSink
[2022-03-18T21:38:01,071][INFO ][o.o.s.a.s.SinkProvider   ] [9b1afcc5a7ba] Closing DebugSink
[2022-03-18T21:38:01,123][INFO ][o.o.n.Node               ] [9b1afcc5a7ba] stopped
[2022-03-18T21:38:01,123][INFO ][o.o.n.Node               ] [9b1afcc5a7ba] closing ...
[2022-03-18T21:38:01,134][INFO ][o.o.s.a.i.AuditLogImpl   ] [9b1afcc5a7ba] Closing AuditLogImpl
[2022-03-18T21:38:01,142][INFO ][o.o.n.Node               ] [9b1afcc5a7ba] closed
Killing performance analyzer process 105
OpenSearch exited with code 143
Performance analyzer exited with code 143

[dlvenable]

These tests are running more consistently now. I do believe it does still fail sometimes.

[dlvenable]

I'm closing this issue as these tests are fairly consistent. We can open a new one if necessary.