[BUG] Opensearch connection refused

[vishnujanardhanan]

Describe the bug I'm trying to set up a monitoring solution on mac using docker opensearch, data prepper and aws distro open telemetry collector. Opensearch comes up and when I start data prepper the below error is thrown. Don't see much help on google, so raising an issue. Please help. I'm attaching my data prepper pipeline xml along. For open search no additional config's - just running the plain vanilla docker image

Caused by: java.net.ConnectException: Connection refused at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:907) ~[data-prepper.jar:1.5.1] at org.opensearch.client.RestClient.performRequest(RestClient.java:301) ~[data-prepper.jar:1.5.1] at org.opensearch.client.RestClient.performRequest(RestClient.java:289) ~[data-prepper.jar:1.5.1] at org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1762) ~[data-prepper.jar:1.5.1] at org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1728) ~[data-prepper.jar:1.5.1] at org.opensearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1696) ~[data-prepper.jar:1.5.1] at org.opensearch.client.ClusterClient.getSettings(ClusterClient.java:119) ~[data-prepper.jar:1.5.1] at com.amazon.dataprepper.plugins.sink.opensearch.index.IndexManager.checkISMEnabled(IndexManager.java:149) ~[data-prepper.jar:1.5.1] at com.amazon.dataprepper.plugins.sink.opensearch.index.IndexManager.checkAndCreateIndexTemplate(IndexManager.java:165) ~[data-prepper.jar:1.5.1] at com.amazon.dataprepper.plugins.sink.opensearch.index.IndexManager.setupIndex(IndexManager.java:160) ~[data-prepper.jar:1.5.1] at com.amazon.dataprepper.plugins.sink.opensearch.OpenSearchSink.initialize(OpenSearchSink.java:105) ~[data-prepper.jar:1.5.1] at com.amazon.dataprepper.plugins.sink.opensearch.OpenSearchSink.(OpenSearchSink.java:89) ~[data-prepper.jar:1.5.1]

Commands used

opensearch

docker run
-p 9200:9200 -p 9600:9600
-e "discovery.type=single-node"
-e "plugins.security.disabled=true"
opensearchproject/opensearch:2.2.0

curl -XGET http://localhost:9200 -u 'admin:admin' --insecure

{ "name" : "307e3f84568c", "cluster_name" : "opensearch", "cluster_uuid" : "TjH1gMngR0Oe9a94tZV0cA", "version" : { "distribution" : "opensearch", "number" : "2.2.0", "build_type" : "tar", "build_hash" : "b1017fa3b9a1c781d4f34ecee411e0cdf930a515", "build_date" : "2022-08-09T02:28:05.169390805Z", "build_snapshot" : false, "lucene_version" : "9.3.0", "minimum_wire_compatibility_version" : "7.10.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "The OpenSearch Project: https://opensearch.org/" } pipeline.txt data-prepper.txt

data-prepper

docker run --name data-prepper
-v /Users/vishnu/git/opentelemetry-poc/javaagent-ex/opentelemetry-java-docs/javaagent/pipeline.yaml:/usr/share/data-prepper/pipelines.yaml
-v /Users/vishnu/git/opentelemetry-poc/javaagent-ex/opentelemetry-java-docs/javaagent/data-prepper.yaml:/usr/s error.txt hare/data-prepper/data-prepper-config.yaml
opensearchproject/data-prepper:latest

running opensearch as a http instance.

error:

[graytaylor0]

Hi @vishnujanardhanan,

Based on your pipeline.txt file, your opensearch sink is misconfigured. If you use the trace_analytics_raw index_type, you cannot specify the index field. When using the index_type as trace_analytics_raw, the index used will always be otel-v1-apm-span. So in this case you will want to remove index from the configuration like this

- opensearch:
        hosts:
          - "http://localhost:9200"
        insecure: true
        username: "admin"
        password: "admin"
        index_type: trace-analytics-raw

If you still get connection refused after making this change, it is likely an issue with your docker setup. In your case, http://localhost:9200 will point to the localhost of the data prepper docker container, rather than your actual localhost. If you are on Windows or Mac, to point to the actual localhost from a docker container you will need to use host.docker.internal. So you can try changing your configuration to

- opensearch:
        hosts:
          - "host.docker.internal:9200"
        insecure: true
        username: "admin"
        password: "admin"
        index_type: trace-analytics-raw

If this doesn't work, we also have a log analytic demo example where data prepper and opensearch are attached to the same docker network, and data prepper can hit the opensearch container directly (https://github.com/opensearch-project/data-prepper/blob/main/examples/log-ingestion/log_ingestion_demo_guide.md)

[vishnujanardhanan]

@graytaylor0 Many thanks for the help. Did the changes and merged all services into 1 docker compose.. I don't see the connection refused error, but i'm not able to login to the opensearch dashboard. I'll try to figure that dashboard auth issue. Attaching the startup logs for you quick review - how to validate whether data prepper has started successfully with opensearch as the sink please? docker-compose-latest.txt logs.txt pipeline-new.txt

[graytaylor0]

Hi @vishnujanardhanan,

To validate that data prepper is running correctly, you should see some logs such as

2021-10-30T12:17:17,474 [log-pipeline-prepper-worker-1-thread-1] INFO  com.amazon.dataprepper.pipeline.ProcessWorker -  log-pipeline Worker: Processing 2 records from buffer

I don't see any of those in the logs.txt you shared. The only data-prepper logs I see are

data-prepper    | 2022-09-07T16:13:03,281 [main] INFO  com.amazon.dataprepper.parser.config.DataPrepperAppConfiguration - Command line args: /usr/share/data-prepper/pipelines.yaml,/usr/share/data-prepper/data-prepper-config.yaml
data-prepper    | 2022-09-07T16:13:03,291 [main] INFO  com.amazon.dataprepper.parser.config.DataPrepperArgs - Using /usr/share/data-prepper/pipelines.yaml configuration file

Which are normal startup logs.

I do see an error from otel-collector, and I believe this is due to a typo in your docker-compose where you set otel-collector container to be dependent on data-prepper. Instead of

    depends_on:
      - dataprepper

I believe it should be

    depends_on:
      - data-prepper

Since that is the container name. I believe this is leading to the following error

awscollector_1  | 2022-09-07T16:12:52.467Z	warn	zapgrpc/zapgrpc.go:191	[core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
awscollector_1  |   "Addr": "host.docker.internal:21890",
awscollector_1  |   "ServerName": "host.docker.internal:21890",
awscollector_1  |   "Attributes": null,
awscollector_1  |   "BalancerAttributes": null,
awscollector_1  |   "Type": 0,
awscollector_1  |   "Metadata": null
awscollector_1  | }. Err: connection error: desc = "transport: authentication handshake failed: EOF"	{"grpc_log": true}
awscollector_1  | 2022-09-07T16:12:52.468Z	warn	zapgrpc/zapgrpc.go:191	[core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
awscollector_1  |   "Addr": "host.docker.internal:21890",
awscollector_1  |   "ServerName": "host.docker.internal:21890",
awscollector_1  |   "Attributes": null,
awscollector_1  |   "BalancerAttributes": null,
awscollector_1  |   "Type": 0,
awscollector_1  |   "Metadata": null
awscollector_1  | }. Err: connection error: desc = "transport: authentication handshake failed: EOF"	{"grpc_log": true}

I don't think this will fix the issue of not being able to log into opensearch though

[vishnujanardhanan]

Dear @graytaylor0, Many thanks for your continuous help on this. I think i am able to get data prepper working based on the below logs

dataprepper | 2022-09-12T15:17:43,854 [main] INFO com.amazon.dataprepper.plugins.sink.opensearch.OpenSearchSink - Initialized OpenSearch sink dataprepper | 2022-09-12T15:17:49,670 [main] INFO com.amazon.dataprepper.pipeline.server.DataPrepperServer - Data Prepper server running at :4900

issue was with docker networking. host.docker.internal didn't work on mac either, but gateway.docker.internal works -> https://aeoluswing.xyz/docker-for-mac/networking/

documenting it here, hope it helps someone :)

[rvsoni]

I am facing the same Caused by: java.net.ConnectException: Connection refused issue in Docker Compose on Linux env. Use case is 1 node Opensearch + Dashboard + Data-prepper.

Date Prepper is not able to connect to Opensearch node.

The same config works well when running all OpenSearch Node + Dashboard on one docker-compose with port exposed from Node, and running Data prepper on a separate docker-compose to connect o OpenSearch node via exposed port works well.

[graytaylor0]

Hi @rvsoni,

Were you able to resolve the issue? If not could you please share your docker-compose files for both Data prepper and OpenSearch. Are you running the data-prepper docker-compose after OpenSearch is already up and running?

[rvsoni]

@graytaylor0

Pl. see the attached AWS OpenSearch and DataPrepper docker-compose sample. OpenSearch-DataPrepper.zip

in this sample

opensearch-dashboards can connect to opensearch-node1 with docker networking but data-prepper is not able to connect and throws a connection refuse error on pipeline loading time.

see docker-compose output

dateprepper.log

[graytaylor0]

@rvsoni While your docker-compose does have the data-prepper container depend on the opensearch-node1 container, it does not wait until the node is completely ready for data prepper to connect to it before starting the data prepper. This is a poor user experience, and is something we are looking to improve for data prepper's opensearch sink in the future. One solution to this issue would be to have one docker-compose start opensearch and opensearch dashboards, and then to start data prepper only after you have verified that opensearch is up and running (this can be done by going to opensearch dasboards at http://localhost:5601). However, to keep you from having to make this change, I have modified your docker-compose.yml file to run a script that waits for opensearch to be completely ready. I have tested this locally and can confirm that it works as intended. Modify the data-prepper container configuration in your docker-compose.yml to be

data-prepper:
    image: opensearchproject/data-prepper:1.5.1
    working_dir: /usr/share/data-prepper/
    command: sh data-prepper-wait-for-opensearch-and-start.sh
    volumes:  
      - ./data-prepper-pipelines.yml:/usr/share/data-prepper/pipelines.yaml
      - ./data-prepper-config.yml:/usr/share/data-prepper/data-prepper-config.yaml
      - ./data-prepper-wait-for-opensearch-and-start.sh:/usr/share/data-prepper/data-prepper-wait-for-opensearch-and-start.sh
    ports:
      - 21890:21890
      - 21891:21891
    expose:
      - "21890"
      - "21891"
    depends_on:
      - opensearch-node1
    networks:
      - opensearch-net

and then create a file in the same directory named data-prepper-wait-for-opensearch-and-start.sh. Copy the following contents into this file

#!/bin/bash

until [[ $(curl --write-out %{http_code} --output /dev/null --silent --head --fail https://opensearch-node1:9200 -u admin:admin --insecure) == 200 ]]; do
  echo "Waiting for OpenSearch to be ready"
  sleep 1
done

java -jar data-prepper.jar pipelines.yaml data-prepper-config.yaml

and you should now be able to run docker-compose up as normal, since data prepper will wait for a successful curl request to write to the cluster before starting. Let me know if you run into any issues with this. Thanks!

[rvsoni]

Thanks, @graytaylor0 for looking into this issue, and also making a workaround,

Currently, I start OpenSearch in one docker-compose first, once it is ready, start data prepper in another docker-compose and use http://localhost:9200 as an OpenSearch connection.

The idle fix would be to make an OpenSearch sink to be lazy on startup and not fail on initialization.

Thanks, Ravi

[dlvenable]

The idle fix would be to make an OpenSearch sink to be lazy on startup and not fail on initialization.

@rvsoni , We do have #936 open to provide this. Feel free to take a look and provide any feedback that you have.

[dlvenable]

@rvsoni, It sounds like @graytaylor0 was able to help you. It seems there is nothing else here so I will close this issue. We do still have #936 open to improve this behavior in a future release.

[rvsoni]

@dlvenable Yes, the suggestion was very helpful to make things work, I would like to wait and test the actual fix in code #936 Thanks,

My demo application is on https://github.com/rvsoni/aws-opensearch-observability-demo

[Gauravkumar123]

@graytaylor0 Please have a look what could be the issue .I am not able to establish connection between opensearch and data-prepper My opensearch dashboard is up and running. i am using docker compose file. I am working in window OS. opensource.txt log_pipeline.txt data-prepper.txt I am able to access your opensearch cluster and opensearch dashboard curl -k https://localhost:9200 -u admin:admin { "name" : "opensearch-node1", "cluster_name" : "opensearch-cluster", "cluster_uuid" : "AtWMvMthRNKqYOMK3pAKbQ", "version" : { "distribution" : "opensearch", "number" : "2.10.0", "build_type" : "tar", "build_hash" : "eee49cb340edc6c4d489bcd9324dda571fc8dc03", "build_date" : "2023-09-20T23:54:29.889267151Z", "build_snapshot" : false, "lucene_version" : "9.7.0", "minimum_wire_compatibility_version" : "7.10.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "The OpenSearch Project: https://opensearch.org/" }