Enable CustomDbtNormalization on k8s

[Grayfados]

Solves the issue 5091 - now, only for bigquery destination.

The purpose of this modifications is to "re-clone" the repo inside the pod if the folder doesn't exists. It reuses the structure of the NormalizationRunner.

At this points it uses a custom script to create the profile of bigquery destination. The better form is to reuse the transform-config script to create the dbt profile (pls someone should help me on that).

What

Describe what the change is solving It helps to add screenshots if it affects the frontend. Enable CustomDbtNormalization on k8s

How

Describe the solution Added the NormalizationRunner steps to DbtNormalizationRunner and enforced the creation of git_repo inside the normalization pod.

Recommended reading order

1. x.java
2. y.java

Can this PR be safely reverted / rolled back?

If you know that your PR is backwards-compatible and can be simply reverted or rolled back, check the YES box.

Otherwise if your PR has a breaking change, like a database migration for example, check the NO box.

If unsure, leave it blank.

• [x] YES 💚
• [] NO ❌

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

[Grayfados]

@davinchia Can this be used?

[elliottrabac]

How can we benefit from this update? We would love to run dbt transformation on k8s.

[anaselmhamdi]

Small bump on this, would love to have this merged !

[franviera92]

read this Airbyte is going to stop using dbt internally as the tool

[5ylar]

read this Airbyte is going to stop using dbt internally as the tool

@franviera92

Is it true? Can you re-check the link of Airbyte is going to stop using dbt internally as the tool? I guess your link is not correct.

[nurikk-sa]

read this Airbyte is going to stop using dbt internally as the tool

@franviera92

Is it true? Can you re-check the link of Airbyte is going to stop using dbt internally as the tool? I guess your link is not correct.

It's for internal usage, dbt invocation and the end of pipeline should still work

https://github.com/airbytehq/airbyte/issues/25194#issuecomment-1680823987

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
0 out of 2 committers have signed the CLA.

:x: Grayfados
:x: pedropinho2022
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[github-actions[bot]]

Your branch is not currently up-to-date with main. Please update your branch before attempting to snapshot your PR.

[marcosmarxm]

This feature was deprecated.

[github-actions[bot]]

Your branch is not currently up-to-date with main. Please update your branch before attempting to snapshot your PR.