Setup Trino Dynamic connectors/catalogs

[ryscheng]

What is it?

It's beta, but worth trying anyway?

[linear[bot]]

OSO-698 Setup Trino Dynamic connectors/catalogs

[ravenac95]

so @ryscheng, @IcaroG and I had a long discussion about this one. Some thoughts from our discussion:

• We realized that depending on when we want to have private connectors changes how we might approach solving the greater problem here.
• Public connectors are simple
• At what granularity do we need these permissions?
  • Will everyone in an org have access if the org has acccess, do we need to guarantee tighter controls?
• Two thoughts on implementation that might change some business things:
  1. Deploy using a separate policy service (called opa) basically allows us to set granular permissions for each connector. We need to spin up and maintain that service. This also requires we do things as dynamic connectors.
  2. Deploy each org to their own "cluster" of trino. If we do it this way technically we could treat the connectors as static for every org. We need to then deploy a trino gateway (to route users to the correct org owned trino cluster) and also we'd need to make sure keda performs correctly. This makes tracking billing on a per user basis quite simple because it's about the cluster. It's maybe a little more work than option 1 but if it's important for us to get a very clear cost per user number this is the easiest way.

We have a little time to make some decisions as we need to get this dynamic connector thing working first (option 2 wouldn't need it but i think it's better than requiring that a connector be available at boot time).

[IcaroG]

Closed in https://github.com/opensource-observer/oso/pull/3741