datahub icon indicating copy to clipboard operation
datahub copied to clipboard

Invalid URNs generated for Metabase Dashboards with Text or Heading Cards

Open pulsar256 opened this issue 9 months ago • 0 comments

Describe the bug

Dashboards with virtual (Text or Heading) Cards cannot be ingested due to invalid URNs.

To Reproduce Steps to reproduce the behavior:

  1. Create a dashboard in Metabase
  2. Add a heading and/or a text element to the dashboard
  3. Run datahub metadata ingestion
  4. Notice errors

Expected behavior

Metadata can be ingested without errors

Screenshots

{'cli_version': 'unavailable (installed in develop mode)',
 'cli_entry_location': '/home/pulsar/development/4screen/datahub/metadata-ingestion/src/datahub/__init__.py',
 'models_version': 'bundled',
 'py_version': '3.10.14 (main, Apr 25 2024, 08:20:49) [GCC 13.2.1 20230801]',
 'py_exec_path': '/home/pulsar/development/4screen/datahub/metadata-ingestion/venv/bin/python',
 'os_details': 'Linux-6.8.7-zen1-1-zen-x86_64-with-glibc2.39',
 'mem_info': '109.77 MB',
 'peak_memory_usage': '109.77 MB',
 'disk_info': {'total': '965.39 GB', 'used': '616.21 GB', 'used_initally': '616.21 GB', 'free': '300.07 GB'},
 'peak_disk_usage': '616.21 GB',
 'thread_count': 4,
 'peak_thread_count': 4}
Source (metabase) report:
{'events_produced': 2,
 'events_produced_per_sec': 3,
 'entities': {'dashboard': ['urn:li:dashboard:(metabase,1)']},
 'aspects': {'dashboard': {'dashboardInfo': 1, 'ownership': 1, 'status': 1}},
 'warnings': {},
 'failures': {},
 'start_time': '2024-04-25 16:39:14.176924 (now)',
 'running_time': '0.56 seconds'}
Sink (datahub-rest) report:
{'total_records_written': 1,
 'records_written_per_second': 1,
 'warnings': [],
 'failures': [{'error': 'Unable to emit metadata to DataHub GMS: com.linkedin.metadata.entity.validation.ValidationException: Failed to validate '
                        'record with class com.linkedin.entity.Entity: ERROR :: '
                        '/value/com.linkedin.metadata.snapshot.DashboardSnapshot/aspects/0/com.linkedin.dashboard.DashboardInfo/charts/0 :: '
                        '"Provided urn urn:li:chart:(metabase,)" is invalid\n'
                        'ERROR :: /value/com.linkedin.metadata.snapshot.DashboardSnapshot/aspects/0/com.linkedin.dashboard.DashboardInfo/charts/1 :: '
                        '"Provided urn urn:li:chart:(metabase,)" is invalid\n',
               'info': {'exceptionClass': 'com.linkedin.restli.server.RestLiServiceException',
                        'message': 'com.linkedin.metadata.entity.validation.ValidationException: Failed to validate record with class '
                                   'com.linkedin.entity.Entity: ERROR :: '
                                   '/value/com.linkedin.metadata.snapshot.DashboardSnapshot/aspects/0/com.linkedin.dashboard.DashboardInfo/charts/0 '
                                   ':: "Provided urn urn:li:chart:(metabase,)" is invalid\n'
                                   'ERROR :: '
                                   '/value/com.linkedin.metadata.snapshot.DashboardSnapshot/aspects/0/com.linkedin.dashboard.DashboardInfo/charts/1 '
                                   ':: "Provided urn urn:li:chart:(metabase,)" is invalid\n',
                        'status': 422,
                        'urn': 'urn:li:dashboard:(metabase,1)'}}],
 'start_time': '2024-04-25 16:39:13.777554 (now)',
 'current_time': '2024-04-25 16:39:14.736834 (now)',
 'total_duration_in_seconds': 0.96,
 'max_threads': 15,
 'gms_version': 'v0.13.2',
 'pending_requests': 0}

Versions

Master / 826c4bff77f93a16da39d9b6b18eb4e4b28e52c5

Additional context

Text and Heading dashboard cards are "virtual cards" and have no valid Id. When DataHub Metadata ingestion tries to construct a URN for that card it will result in invalid data: urn:li:chart:(metabase,)

pulsar256 avatar Apr 25 '24 14:04 pulsar256