Open Metadata only ingest last successful run from dbt cloud

[LeoLuo123]

Affected module dbt run result Ingestion Framework

Describe the bug Ingestion workflow only looks up last successful dbt run run_results.json file. It will not ingest run_results.json if dbt run failed no matter it's due to dbt tests failure or something else

To Reproduce

1. Have a dbt job configured on dbt cloud, including a failing test.
2. Start the dbt ingestion workflow by pulling results from dbt cloud.

Expected behavior Ingestion should ingest dbt run_results.json no matter it's successful or failed.

Version:

• OS: [e.g. iOS]
• Python version: 3.8
• OpenMetadata version: [e.g. 0.8] 1.5.0
• OpenMetadata Ingestion package version: [e.g. openmetadata-ingestion[docker]==XYZ] 1.5.0

Additional context Add any other context about the problem here.

The source code: https://github.com/open-metadata/OpenMetadata/blob/42eea35aff60b07fcd574c7964bff5aa212a1aa9/ingestion/src/metadata/ingestion/source/database/dbt/dbt_config.py#L181-L211

Official doc: https://docs.open-metadata.org/latest/connectors/ingestion/workflows/dbt/run-dbt-workflow-externally#6.-dbt-cloud