Problem with analytics when using ES OSS

[murzindima]

Describe the bug I have deployed DataHub using managed ElasticSearch in AWS and for some reason Analytics is not working (http://localhost:9002/analytics).

To Reproduce Steps to reproduce the behavior:

1. Go to 'https://datahubproject.io/docs/quickstart/'
2. Use 'https://github.com/datahub-project/datahub/blob/master/docker/quickstart/docker-compose-without-neo4j-m1.quickstart.yml'
3. Change ES image to 'https://www.docker.elastic.co/r/elasticsearch/elasticsearch-oss:7.9.3' + USE_AWS_ELASTICSEARCH=true
4. See error

Expected behavior A clear and concise description of what you expected to happen.

Screenshots

Additional context ES: managed ES OSS in AWS

{
  "name" : "123qwe123qwe",
  "cluster_name" : "blahblahblah",
  "cluster_uuid" : "123we123qwe",
  "version" : {
    "number" : "7.10.2",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "unknown",
    "build_date" : "2022-02-10T09:41:23.620550Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

PostgreSQL: PostgreSQL 14.3 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44), 64-bit DataHub: v0.8.40

Logs: tmpnqxgvxye.log

[tomas-kubin]

Hello @murzindima, I have encountered a similar issue. This is my workaround. Please let me know if it helped you. If so, I will submit a possible fix so that the workaround is not needed anymore.

Workaround

• Go to Kibana console (link should be available in AWS OpenSearch domain detail.)
• Execute the following three queries one by one. Maybe some of the will fail. If so, please let me know:

PUT _template/datahub_usage_event_index_template
{
  "index_patterns": ["datahub_usage_event-*"],
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "type": {
        "type": "keyword"
      },
      "timestamp": {
        "type": "date"
      },
      "userAgent": {
        "type": "keyword"
      },
      "browserId": {
        "type": "keyword"
      }
    }
  },
  "settings": {
    "index.opendistro.index_state_management.rollover_alias": "datahub_usage_event"
  }
}

DELETE datahub_usage_event

PUT datahub_usage_event-000001
{
  "aliases": {
    "datahub_usage_event": {
      "is_write_index": true
    }
  }
}

[murzindima]

Hello @tomas-kubin

Many thanks! It works like a charm.

[tomas-kubin]

Thanks for the feedback @murzindima ! Now that I have the issue confirmed from two sources, I will try to make a proper fix.

[anshbansal]

@tomas-kubin If you do make a contribution that would be great. Please head to #contribute channel in slack in case you need any help

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity. If you believe this is still an issue on the latest DataHub release please leave a comment with the version that you tested it with. If this is a question/discussion please head to https://slack.datahubproject.io. For feature requests please use https://feature-requests.datahubproject.io

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity. If you believe this is still an issue on the latest DataHub release please leave a comment with the version that you tested it with. If this is a question/discussion please head to https://slack.datahubproject.io. For feature requests please use https://feature-requests.datahubproject.io

[aditya-radhakrishnan]

Closing this issue since it has been resolved. If you'd like to make a contribution to fix this, that'd be great. Thank you!