datahub icon indicating copy to clipboard operation
datahub copied to clipboard

Athena views not retrieving definition and lineage

Open KulykDmytro opened this issue 2 years ago • 0 comments

Describe the bug When extracting Athena views over standard ingest - no SQL definitions and upstream tables lineage being ingested

To Reproduce Steps to reproduce the behavior:

  1. setup Athena ingest file like this
source:
  type: athena
  config:
    aws_region: eu-central-1
    s3_staging_dir: s3://athena-result-bucket/results
    work_group: primary
    include_views: true
    include_tables: false
    schema_pattern:
      deny:
        - inc
        - tmp
        - sandbox
        - sampledb
        - ^region_.*
transformers:
  - type: simple_add_dataset_tags
    config:
      tag_urns:
        - "urn:li:tag:AWS"
sink:
  type: datahub-rest
  config:
    server: http://datahub-gms:8080
  1. ingest source
  2. observe view with reduced aspects set

Expected behavior Athena view should have definition (SQL) and upstream tables lineage

Screenshots image image

Additional context EKS - based deployment in same Account view helm chart

KulykDmytro avatar Aug 09 '22 18:08 KulykDmytro

This issue is stale because it has been open for 30 days with no activity. If you believe this is still an issue on the latest DataHub release please leave a comment with the version that you tested it with. If this is a question/discussion please head to https://slack.datahubproject.io. For feature requests please use https://feature-requests.datahubproject.io

github-actions[bot] avatar Sep 11 '22 02:09 github-actions[bot]