[BUG] Pipeline [raw-pipeline] - Workers did not terminate in time, forcing termination

[wwwlll2001]

Describe the bug Data prepper did not process tracing silently and I find the log below in the pod log, I suspend this is the cause, but do not know how to fix it

2022-04-12T15:03:07,141 [raw-pipeline-sink-worker-6-thread-1] WARN com.amazon.dataprepper.pipeline.Pipeline - Pipeline [raw-pipeline] - Workers did not terminate in time, forcing termination

because after this time point, the tracing disappears in my opensearch tracing GUI, and also otel-collector log started to show that it could not connect to data-prepper as "connection refused" appeared in its log

To Reproduce Steps to reproduce the behavior: 1.Just regular usage, otel java agent --- otel-collector pod --- data-prepper pod --- opensearch

Expected behavior Data prepper could work normally instead of stop working siliently

Environment (please complete the following information): image :opensearchproject/data-prepper:1.3.0

[asifsmohammed]

@wwwlll2001 Thanks for reporting the bug. I just have few questions before trying to reproduce the bug,

1. Have you tried shutting down Data Prepper or is it terminated without any action?
2. Can you provide us the logs before shutdown to understand what's really happening if you still have the logs? It would be even better if you could share your configuration.

[wwwlll2001]

@asifsmohammed Thanks for your kindly response

1. I have not tried to shut down data prepper, it's just forced to be terminated without any action
2. Sorry, I just copy the log which report the terminating info, and I could share my configuration as below

apiVersion: v1
kind: ConfigMap
metadata:
  name: pipelines-conf
  labels:
    app: opentelemetry
    component: pipelines-conf
data:
  pipelines: |
    entry-pipeline:
      delay: "100"
      source:
        otel_trace_source:
          ssl: false
      buffer:
        bounded_blocking:
          buffer_size: 128
          batch_size: 8
      sink:
        - pipeline:
            name: "raw-pipeline"
        - pipeline:
            name: "service-map-pipeline"
    raw-pipeline:
      source:
        pipeline:
          name: "entry-pipeline"
      prepper:
        - otel_trace_raw_prepper:
      sink:
        - opensearch:
            hosts: {{ .Values.opensearch.hosts}}
            insecure: true
            username: {{ .Values.opensearch.username}}
            password: {{ .Values.opensearch.password}}
            trace_analytics_raw: true
    service-map-pipeline:
      delay: "100"
      source:
        pipeline:
          name: "entry-pipeline"
      prepper:
        - service_map_stateful:
      sink:
        - opensearch:
            hosts: {{ .Values.opensearch.hosts}}
            insecure: true
            username: {{ .Values.opensearch.username}}
            password: {{ .Values.opensearch.password}}
            trace_analytics_service_map: true   

[asifsmohammed]

@wwwlll2001 Thanks for providing the configuration.

Unfortunately, we aren't able to reproduce the termination of Data Prepper but we'll try to find if there's any issue why the shutdown method is called. Please reach out to us if this happens again and try to provide the logs which will help us resolve the issue.