Startup probe and Readiness probe failed.

[NoorKumar]

I have installed the operator and trying to create cluster with the example config given. Its creating the pods but pods are not in ready state as startup probe is failing. Startup probe failed: dial tcp 1.x.x.x:9200: connect: connection refused

I tried to exec one of the pod and executed curl commands, below is the result.

sh-4.2$ curl -XGET https://localhost:9200 -u 'admin:admin' --insecure OpenSearch Security not initialized.sh-4.2$ ^C sh-4.2$ ^C sh-4.2$ ^C sh-4.2$ curl -XGET http://localhost:9200 -u 'admin:admin' --insecure curl: (52) Empty reply from server sh-4.2$

Can anyone help me to understand what is going wrong.

apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: my-cluster
  namespace: opensearch-demo
spec:
  general:
    version: 1.3.2
    httpPort: 9200
    vendor: opensearch
    serviceName: my-cluster
  dashboards:
    version: 1.3.0
    enable: true
    replicas: 2
    resources:
      requests:
         memory: "1Gi"
         cpu: "500m"
      limits:
         memory: "1Gi"
         cpu: "500m"
  confMgmt:
    smartScaler: true
  nodePools:
    - component: masters
      replicas: 3
      diskSize: "1Gi"
      NodeSelector:
      resources:
         requests:
            memory: "2Gi"
            cpu: "500m"
         limits:
            memory: "2Gi"
            cpu: "500m"
      roles:
        - "master"
        - "data"
    - component: nodes
      replicas: 3
      diskSize: "1Gi"
      NodeSelector:
      resources:
         requests:
            memory: "2Gi"
            cpu: "500m"
         limits:
            memory: "2Gi"
            cpu: "500m"
      roles:
        - "data"
    - component: coordinators
      replicas: 3
      diskSize: "1Gi"
      NodeSelector:
      resources:
         requests:
            memory: "2Gi"
            cpu: "500m"
         limits:
            memory: "2Gi"
            cpu: "500m"
      roles:
        - "ingest"

Below is the opensearch.yml in the pod

cluster.name: docker-cluster

#### Bind to all interfaces because we don't know what IP address Docker will assign to us. ###
network.host: 0.0.0.0

###  minimum_master_nodes need to be explicitly set when bound on a public IP ###
###  set to 1 to allow single node clusters ###
### discovery.zen.minimum_master_nodes: 1 ###

### Setting network.host to a non-loopback address enables the annoying bootstrap checks. "Single-node" mode disables them again. ###
### discovery.type: single-node ###

######## Start OpenSearch Security Demo Configuration ########
### WARNING: revise all the lines below before you go into production
plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - CN=kirk,OU=client,O=client,L=test, C=de

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*",
".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".opendistro-asynchronous-
search-response*", ".replication-metadata-store"]
node.max_local_storage_nodes: 3
######## End OpenSearch Security Demo Configuration ########

[swoehrl-mw]

Hi @NoorKumar, can you check in kubernetes if the pod <cluster-name>-securityconfig-update is completed and what its log says? You could also check the log of the opensearch pods themselves if they report any problems. Note that starting up a cluster can take a few minutes. Also: Did you test with the newest version (2.0)?

[swoehrl-mw]

Closing, as there was no further response from the issue reporter.