OpenMetadata icon indicating copy to clipboard operation
OpenMetadata copied to clipboard

Fix #21679: Added `metadata ingest-dbt` CLI Command for Direct DBT Artifacts Ingestion

Open Khairajani opened this issue 5 months ago • 10 comments

Fix #21679: Added metadata ingest-dbt CLI Command for Direct DBT Artifacts Ingestion

Description:

This PR introduces a new CLI command metadata ingest-dbt that enables direct ingestion of dbt artifacts to OpenMetadata without requiring separate workflow configuration files. This streamlines the dbt metadata ingestion process by reading configuration directly from the dbt project's dbt_project.yml file.

📋 Usage:

# If in DBT project directory:
metadata ingest-dbt

# If in a different directory:
metadata ingest-dbt -c /path/to/dbt-project

⚙️ Configuration:

Users add OpenMetadata configuration directly to their dbt_project.yml:

# dbt_project.yml

vars:
  # OpenMetadata configuration for metadata ingest-dbt command
  openmetadata_host_port: "https://your-openmetadata-url/your-endpoint" #(e.g.: http://localhost:8585/api)
  openmetadata_jwt_token: "your-jwt-token" # (or set environment variable using -  ${OPENMETADATA_JWT_TOKEN})
  openmetadata_service_name: "your-service-name"

🧪 Test Coverage:

The comprehensive test suite validates:

  • dbt_project.yml parsing and vars section validation
  • OpenMetadata configuration extraction with all edge-case scenarios
  • Variable format validation (naming conventions, URL formats)
  • CLI execution and workflow configuration creation

Type of change:

  • [ ] Bug fix
  • [ ] Improvement
  • [x] New feature
  • [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • [ ] Documentation

Checklist:

  • [x] I have read the CONTRIBUTING document.
  • [x] My PR title is Fixes <issue-number>: <short explanation>
  • [x] I have commented on my code, particularly in hard-to-understand areas.
  • [ ] For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

New feature

  • [x] The issue properly describes why the new feature is needed, what's the goal, and how we are building it. Any discussion or decision-making process is reflected in the issue.
  • [ ] I have updated the documentation.
  • [x] I have added tests around the new logic.

Khairajani avatar Jun 10 '25 10:06 Khairajani