Use nessie's branching for sqlmesh writes

[ravenac95]

What is it?

Running sqlmesh can sometimes be A disruptive, B (worse) destructive and erroneous. This is likely due to some random infrastructure errors here and there. So what we should instead do is utilize nessie's branching to allow for us to safely roll-forward/backward into "production".

I propose we do the following (I'm assuming this all works):

• SQLMesh via Producer Trino always writes to main (whatever the nessie equivalent) branch. We do this so that sqlmesh constantly writes to the same place as it assumes it always will.
• We have a "release promotion" job in dagster. This will do 2 things
  • Create a nessie branch (or tag) from the main branch
  • Create a git commit (or PR) to automatically update the release version for consumer trino
• Rollback is simply choosing a different release branch/tag

[linear[bot]]

OSO-579 Use nessie's branching for sqlmesh writes

[ryscheng]

@ravenac95 @IcaroG I think we ended up doing branching with sqlmesh environments rather than Nessie branching right?

Is this issue still relevant? Is this a better way to handle branching? Are we positive that the sqlmesh environment approach that we took is sufficiently safe?

[ryscheng]

@ravenac95 suggests we still do this for consumer vs producer trino

[ravenac95]

The proposed workflow has now changed and I think this will give us the most flexibility and safety.

[ccerv1]

We should make it easy for any internal analysts to access the most up-to-date branch of producer trino. Not sure exactly how we want to do this yet, but it might be a "private connector" in the oso org or perhaps a different "internal-consumer" trino deployment.

Are you envisioning something like:

from pyoso import Client
client = Client(api_key=OSO_API_KEY, branch='producer') # default branch='consumer'