QHub Extension Mechanism

Open costrouc opened this issue 2 years ago • 11 comments

Description

The are many projects that we would like to integrate into QHub and additionally other projects that we may not have the time/resources to support in QHub. We need an extension mechanism to integrate projects that a loosely connected to Qhub. For me prefect and clearml are good examples of this. I would also like to see if this could be adopted within Qhub for some parts.

Suggestion

Features that the extension mechanism should have.

allows us to expose the service at https:////
optionally setup oauth2 client for service
provide an "overrides" section so that we can easily control the helm chart
helm chart information

Example

extensions:
  - name: Prefect
    chart: <path-to-chart-url>
    version: <chart-version>
    domain: <optional domain name>
    prefix: <prefix-url>
    oauth2_client: true/false
    overrides:
       ...

Lets start a discussion about this. We need to use approach as well for a few components.

Oct 15 '21 16:10 costrouc

Let me talk in detail a bit more about these pieces.

OAuth2 Provisioning

We should provision an OAuth2 client per service with keycloak this is possible an encouraged.

Should take as inputs:

callback_url which should be roughly calculated from domain/prefix.

This should return:

endpoint
user url information
client id
secret

This is complex ... not sure how to expose to the helm chart. Possibly have some way to reference these variables within the overrides? Ideas welcome.

Ingress Provisioning

Typically helm charts already have ingress hooks. How do we make sure that we are integrating this properly. I'd prefer if we hook into the existing chart for this. Since traefik will do the work for provisioning certs. Need to think about this more.

Monitoring Integration

We should apply annotations that allow prometheus monitoring to easily connect.

Oct 15 '21 16:10 costrouc

As a first crack at this, I'd like to implement the simple bits below. Any problem with merging something like what's below in as a first step? @costrouc

extensions:
  - name: <helm-deployment-name>
    chart: <path-to-chart-url>
    version: <chart-version>
    overrides:
       ...

Oct 20 '21 18:10 Adam-D-Lewis

I like it @Adam-D-Lewis and nice in that it is definitely the minimum that would be useful and would not introduce complex dependencies on QHub bits.

Oct 20 '21 18:10 costrouc

@Adam-D-Lewis and @costrouc

Please note I already have an extensions key as part of the Keycloak branch, as in #751

It doesn't accept helm charts, and in any case needs some consideration for its design, but it would definitely conflict given the proposal above!

Oct 21 '21 11:10 danlester

I have a working basic version (not pushed yet). I'm playing around trying to deploy some helm charts with the proposed changes.

Made some progress on deploying a prefect server helm chart. It seems I'm experiencing and issue similar to https://stackoverflow.com/questions/53752270/traefik-path-based-routing. It appears that when I try to go to http://github-actions.qhub.dev/prefect-ui, all the sublinks in /var/www/index.html on the prefect-ui pod are directing the browser to look for resources from e.g. github-actions.qhub.dev/fonts/fonts.css instead of github-actions.qhub.dev/prefect-ui/fonts/fonts.css as it should so we're getting 404 errors when loading github-actions.qhub.dev/prefect-ui and it doesn't ever fully load. Instead, we get a loading bar that gets stuck (see image)

I've also included the index.html from the prefect-ui pod below

<!-- /var/www/index.html on prefect-ui pod -->
<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"><meta http-equiv="Content-Security-Policy" content="
    default-src *;
    script-src 'self' 'unsafe-eval' 'unsafe-inline' https://cdn.lr-ingest.io https://js.stripe.com;
    img-src * data:;
    style-src-elem 'self' 'unsafe-inline';
    script-src-elem 'self' 'unsafe-inline' https://cdn.lr-ingest.io https://js.stripe.com https://netlify-cdp-loader.netlify.app/netlify.js;
    style-src 'self' 'unsafe-inline';
    font-src data: 'self';
    frame-src https://*.prefect.io https://prefect.auth0.com https://js.stripe.com https://cloud.prefect.io https://*.okta.com;
    child-src data: blob:;
    worker-src data: blob: 'self';
  "><meta http-equiv="X-UA-Compatible" content="IE=edge"><meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1,user-scalable=no"><link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon.png"><link rel="icon" type="image/png" sizes="32x32" href="/favicon-32x32.png"><link rel="icon" type="image/png" sizes="16x16" href="/favicon-16x16.png"><link rel="manifest" href="/site.webmanifest"><link rel="mask-icon" href="/safari-pinned-tab.svg" color="#5bbad5"><meta name="msapplication-TileColor" content="#2d89ef"><meta name="theme-color" content="#ffffff"><title>Prefect</title><link rel="stylesheet" href="/fonts/fonts.css"><script defer src="/fonts/font-awesome.min.js"></script><script src="https://js.stripe.com/v3/" async></script><link href="/css/Auth~f71cff67.47831603.css" rel="prefetch"><link href="/css/accept~b07b7304.e1a9939b.css" rel="prefetch"><link href="/css/access-denied~56ca29ff.0a770553.css" rel="prefetch"><link href="/css/admin--account~31ecd969.692ca8c2.css" rel="prefetch"><link href="/css/admin--new-team~21833f8f.f559c431.css" rel="prefetch"><link href="/css/admin--teams-overview~21833f8f.25c54fbb.css" rel="prefetch"><link href="/css/admin--teams~72c2ec36.492c7c74.css" rel="prefetch"><link href="/css/admin~31ecd969.55725814.css" rel="prefetch"><link href="/css/agents~49518580.2b8f32d9.css" rel="prefetch"><link href="/css/agent~31ecd969.b5cc3978.css" rel="prefetch"><link href="/css/calendar~21833f8f.380818e0.css" rel="prefetch"><link href="/css/chunk-110350d3.2430c35c.css" rel="prefetch"><link href="/css/chunk-2854ddcc.ce6a4d04.css" rel="prefetch"><link href="/css/chunk-f57df11e.1383b760.css" rel="prefetch"><link href="/css/defaultVendors~admin--account~agent~agents~flow~flow-run~project~task~task-run~b5906859.747dabfb.css" rel="prefetch"><link href="/css/defaultVendors~admin~flow~flow-run~getting-started~project~task~task-run~team-settings--members~team~87271691.cb50559a.css" rel="prefetch"><link href="/css/defaultVendors~agents~690b702c.6b4496b5.css" rel="prefetch"><link href="/css/defaultVendors~agent~flow~flow-run~project~task~task-run~team-settings--cloud-hooks~team-settings--f~af47881b.7c4712ab.css" rel="prefetch"><link href="/css/defaultVendors~calendar~dde583c9.29ce509a.css" rel="prefetch"><link href="/css/defaultVendors~calendar~flow~flow-run~project~task-run~team-settings--service-accounts~user-settings~0920559e.6aa76c4d.css" rel="prefetch"><link href="/css/defaultVendors~notifications~dde583c9.18966a5d.css" rel="prefetch"><link href="/css/defaultVendors~task~fdc6512a.1bcf339c.css" rel="prefetch"><link href="/css/defaultVendors~team-settings--cloud-hooks~dde583c9.2dbfc10d.css" rel="prefetch"><link href="/css/defaultVendors~team-settings--members~690b702c.747dabfb.css" rel="prefetch"><link href="/css/defaultVendors~user-settings--keys~690b702c.ca6df252.css" rel="prefetch"><link href="/css/default~admin--account~plans~21833f8f.1b3b83d5.css" rel="prefetch"><link href="/css/default~agent~agents~21833f8f.1134b59e.css" rel="prefetch"><link href="/css/default~team-settings--kv~team-settings--secrets~31ecd969.615fa4c8.css" rel="prefetch"><link href="/css/default~team-settings--service-accounts~user-settings--keys~21833f8f.058534f7.css" rel="prefetch"><link href="/css/flow-run~21833f8f.0a0fc496.css" rel="prefetch"><link href="/css/flow-run~e2550e02.28fd7ac5.css" rel="prefetch"><link href="/css/flow-run~ec8c427e.3895be2e.css" rel="prefetch"><link href="/css/flow~21833f8f.362cf8b3.css" rel="prefetch"><link href="/css/flow~3d9b8e9e.f64b289b.css" rel="prefetch"><link href="/css/flow~7274e1de.ec54e134.css" rel="prefetch"><link href="/css/flow~7d359b94.111fefe1.css" rel="prefetch"><link href="/css/forgot-password~5a11b65b.7c98f38c.css" rel="prefetch"><link href="/css/getting-started~31ecd969.88163c70.css" rel="prefetch"><link href="/css/interactive-api~3d9b8e9e.b274da9b.css" rel="prefetch"><link href="/css/login~f71cff67.e1ac9b86.css" rel="prefetch"><link href="/css/not-found~b14dea4d.0d39c58d.css" rel="prefetch"><link href="/css/notifications~f71cff67.152d2605.css" rel="prefetch"><link href="/css/onboard--name-team~b07b7304.b99e6a1e.css" rel="prefetch"><link href="/css/onboard--resources~31ecd969.d229d40a.css" rel="prefetch"><link href="/css/onboard--welcome~f71cff67.c7e13c46.css" rel="prefetch"><link href="/css/onboard~3d9b8e9e.981538fb.css" rel="prefetch"><link href="/css/plans~31ecd969.8c0952be.css" rel="prefetch"><link href="/css/project~3d9b8e9e.5f352f57.css" rel="prefetch"><link href="/css/project~7274e1de.ec54e134.css" rel="prefetch"><link href="/css/project~bb9f020d.a037a35d.css" rel="prefetch"><link href="/css/project~ec8c427e.a16db6c4.css" rel="prefetch"><link href="/css/sign-up~f71cff67.e3d85910.css" rel="prefetch"><link href="/css/support~31ecd969.21504d56.css" rel="prefetch"><link href="/css/task-run~21833f8f.1937beb6.css" rel="prefetch"><link href="/css/task-run~e2550e02.60037022.css" rel="prefetch"><link href="/css/task-run~ec8c427e.3895be2e.css" rel="prefetch"><link href="/css/task~21833f8f.5424e8d2.css" rel="prefetch"><link href="/css/team-settings--cloud-hooks~21833f8f.e7084b99.css" rel="prefetch"><link href="/css/team-settings--flow-concurrency~21833f8f.33a72ad9.css" rel="prefetch"><link href="/css/team-settings--flow-groups~21833f8f.f118e5e8.css" rel="prefetch"><link href="/css/team-settings--kv~b07b7304.058534f7.css" rel="prefetch"><link href="/css/team-settings--members~21833f8f.3449eb57.css" rel="prefetch"><link href="/css/team-settings--projects~21833f8f.e25b1a22.css" rel="prefetch"><link href="/css/team-settings--secrets~b07b7304.387f169d.css" rel="prefetch"><link href="/css/team-settings--service-accounts~31ecd969.a74ae44e.css" rel="prefetch"><link href="/css/team-settings--task-concurrency~21833f8f.8fa2fb44.css" rel="prefetch"><link href="/css/team-settings--tokens~21833f8f.1309a66e.css" rel="prefetch"><link href="/css/team-settings~b07b7304.868ae4c6.css" rel="prefetch"><link href="/css/team-switched~f1126255.92949947.css" rel="prefetch"><link href="/css/tutorials~3d9b8e9e.816dda46.css" rel="prefetch"><link href="/css/user-settings--keys~712cfaab.dc76387f.css" rel="prefetch"><link href="/css/user-settings--profile~b07b7304.058534f7.css" rel="prefetch"><link href="/css/user-settings--teams~21833f8f.058534f7.css" rel="prefetch"><link href="/css/user-settings--tokens~21833f8f.969be86c.css" rel="prefetch"><link href="/css/user-settings~3bf82df3.7c0efe5f.css" rel="prefetch"><link href="/js/Auth~f71cff67.91c713a0.js" rel="prefetch"><link href="/js/accept~b07b7304.7c30ab40.js" rel="prefetch"><link href="/js/access-denied~56ca29ff.b6c28bfe.js" rel="prefetch"><link href="/js/admin--account~31ecd969.facb82cf.js" rel="prefetch"><link href="/js/admin--new-team~21833f8f.f508adce.js" rel="prefetch"><link href="/js/admin--teams-overview~21833f8f.516d70de.js" rel="prefetch"><link href="/js/admin--teams~72c2ec36.9a6b11e3.js" rel="prefetch"><link href="/js/admin~31ecd969.81569d78.js" rel="prefetch"><link href="/js/agents~49518580.a2dff007.js" rel="prefetch"><link href="/js/agent~31ecd969.c5b2966a.js" rel="prefetch"><link href="/js/calendar~21833f8f.b293e959.js" rel="prefetch"><link href="/js/chunk-110350d3.4939e285.js" rel="prefetch"><link href="/js/chunk-2854ddcc.71ee0d4f.js" rel="prefetch"><link href="/js/chunk-f57df11e.0c0947a7.js" rel="prefetch"><link href="/js/defaultVendors~admin--account~agent~agents~flow~flow-run~project~task~task-run~b5906859.3187c42b.js" rel="prefetch"><link href="/js/defaultVendors~admin~flow~flow-run~getting-started~project~task~task-run~team-settings--members~team~87271691.381a221b.js" rel="prefetch"><link href="/js/defaultVendors~agents~690b702c.57170457.js" rel="prefetch"><link href="/js/defaultVendors~agent~flow~flow-run~project~task~task-run~team-settings--cloud-hooks~team-settings--f~af47881b.1956db02.js" rel="prefetch"><link href="/js/defaultVendors~calendar~dde583c9.b1bc28a5.js" rel="prefetch"><link href="/js/defaultVendors~calendar~flow~flow-run~project~task-run~team-settings--service-accounts~user-settings~0920559e.1c560bc8.js" rel="prefetch"><link href="/js/defaultVendors~flow~flow-run~project~task~task-run~team-settings--kv~team-settings--secrets~6ab0ef1a.b861ff70.js" rel="prefetch"><link href="/js/defaultVendors~interactive-api~9c5b28f6.5863cc33.js" rel="prefetch"><link href="/js/defaultVendors~interactive-api~b5906859.640be9de.js" rel="prefetch"><link href="/js/defaultVendors~interactive-api~bb6aa01a.1027f005.js" rel="prefetch"><link href="/js/defaultVendors~interactive-api~team-settings--kv~team-settings--secrets~dcae4f2f.10bcaa14.js" rel="prefetch"><link href="/js/defaultVendors~notifications~dde583c9.3e971c1c.js" rel="prefetch"><link href="/js/defaultVendors~notifications~tutorials~1f20a385.40264644.js" rel="prefetch"><link href="/js/defaultVendors~notifications~tutorials~2e5e1226.51beeaaf.js" rel="prefetch"><link href="/js/defaultVendors~notifications~tutorials~690b702c.62315589.js" rel="prefetch"><link href="/js/defaultVendors~notifications~tutorials~7274e1de.3ed2b879.js" rel="prefetch"><link href="/js/defaultVendors~notifications~tutorials~910d3827.81a8f72f.js" rel="prefetch"><link href="/js/defaultVendors~notifications~tutorials~981bc102.de7f1d18.js" rel="prefetch"><link href="/js/defaultVendors~notifications~tutorials~defe4c64.af114b0c.js" rel="prefetch"><link href="/js/defaultVendors~notifications~tutorials~ec8c427e.0251534e.js" rel="prefetch"><link href="/js/defaultVendors~task~fdc6512a.d15ad253.js" rel="prefetch"><link href="/js/defaultVendors~team-settings--cloud-hooks~dde583c9.a3ed4e3d.js" rel="prefetch"><link href="/js/defaultVendors~team-settings--members~690b702c.04814e31.js" rel="prefetch"><link href="/js/defaultVendors~user-settings--keys~690b702c.767e9b58.js" rel="prefetch"><link href="/js/default~admin--account~plans~21833f8f.174b52e4.js" rel="prefetch"><link href="/js/default~agent~agents~21833f8f.9c13288e.js" rel="prefetch"><link href="/js/default~team-settings--kv~team-settings--secrets~31ecd969.ac2bed7f.js" rel="prefetch"><link href="/js/default~team-settings--service-accounts~user-settings--keys~21833f8f.50d20bfc.js" rel="prefetch"><link href="/js/flow-run~21833f8f.107a2335.js" rel="prefetch"><link href="/js/flow-run~2e5e1226.7757250e.js" rel="prefetch"><link href="/js/flow-run~7274e1de.09b9b691.js" rel="prefetch"><link href="/js/flow-run~7d359b94.2912c308.js" rel="prefetch"><link href="/js/flow-run~910d3827.b8193f82.js" rel="prefetch"><link href="/js/flow-run~981bc102.118f1d62.js" rel="prefetch"><link href="/js/flow-run~9c5b28f6.a714e89c.js" rel="prefetch"><link href="/js/flow-run~defe4c64.9d3d4268.js" rel="prefetch"><link href="/js/flow-run~e2550e02.8940e769.js" rel="prefetch"><link href="/js/flow-run~ec8c427e.c870e229.js" rel="prefetch"><link href="/js/flow-run~f9129949.a440d59b.js" rel="prefetch"><link href="/js/flow~1f20a385.d3d60644.js" rel="prefetch"><link href="/js/flow~21833f8f.ea9dff83.js" rel="prefetch"><link href="/js/flow~2e5e1226.2485c400.js" rel="prefetch"><link href="/js/flow~3d9b8e9e.4f16db6a.js" rel="prefetch"><link href="/js/flow~7274e1de.9819f278.js" rel="prefetch"><link href="/js/flow~7d359b94.a99f9821.js" rel="prefetch"><link href="/js/flow~910d3827.672207e3.js" rel="prefetch"><link href="/js/flow~981bc102.28365af4.js" rel="prefetch"><link href="/js/flow~db300d2f.990b6fa1.js" rel="prefetch"><link href="/js/flow~defe4c64.542cf674.js" rel="prefetch"><link href="/js/flow~e2550e02.37256564.js" rel="prefetch"><link href="/js/forgot-password~5a11b65b.7ed8a80d.js" rel="prefetch"><link href="/js/getting-started~31ecd969.2f4e3ba9.js" rel="prefetch"><link href="/js/interactive-api~3d9b8e9e.f6899826.js" rel="prefetch"><link href="/js/login~f71cff67.bf14c0ef.js" rel="prefetch"><link href="/js/not-found~b14dea4d.38a7a6b2.js" rel="prefetch"><link href="/js/notifications~f71cff67.adadd186.js" rel="prefetch"><link href="/js/onboard--name-team~b07b7304.50c23736.js" rel="prefetch"><link href="/js/onboard--resources~31ecd969.eb96b6f6.js" rel="prefetch"><link href="/js/onboard--welcome~f71cff67.17cf42de.js" rel="prefetch"><link href="/js/onboard~3d9b8e9e.aad25cb1.js" rel="prefetch"><link href="/js/plans~31ecd969.e7f5a94e.js" rel="prefetch"><link href="/js/project~1f20a385.e62fa25b.js" rel="prefetch"><link href="/js/project~2e5e1226.c8d941f1.js" rel="prefetch"><link href="/js/project~3d9b8e9e.5da0ad3c.js" rel="prefetch"><link href="/js/project~690b702c.72295737.js" rel="prefetch"><link href="/js/project~7274e1de.45d13548.js" rel="prefetch"><link href="/js/project~910d3827.4543f8be.js" rel="prefetch"><link href="/js/project~981bc102.ade31dce.js" rel="prefetch"><link href="/js/project~b07b7304.666269fe.js" rel="prefetch"><link href="/js/project~bb9f020d.2fa93a7c.js" rel="prefetch"><link href="/js/project~db300d2f.5bc125ee.js" rel="prefetch"><link href="/js/project~defe4c64.616b17ef.js" rel="prefetch"><link href="/js/project~ec8c427e.02be5da7.js" rel="prefetch"><link href="/js/sign-up~f71cff67.6c80dbb5.js" rel="prefetch"><link href="/js/support~31ecd969.744b7c25.js" rel="prefetch"><link href="/js/task-run~21833f8f.8ee28638.js" rel="prefetch"><link href="/js/task-run~2e5e1226.772d0b8c.js" rel="prefetch"><link href="/js/task-run~7274e1de.a11f37eb.js" rel="prefetch"><link href="/js/task-run~7d359b94.cd21e4f8.js" rel="prefetch"><link href="/js/task-run~910d3827.6d8bab8f.js" rel="prefetch"><link href="/js/task-run~981bc102.c9378af5.js" rel="prefetch"><link href="/js/task-run~9c5b28f6.e7437f0d.js" rel="prefetch"><link href="/js/task-run~defe4c64.ba47bfea.js" rel="prefetch"><link href="/js/task-run~e2550e02.9a63ac52.js" rel="prefetch"><link href="/js/task-run~ec8c427e.ee1d2a9a.js" rel="prefetch"><link href="/js/task-run~f9129949.4ed827c8.js" rel="prefetch"><link href="/js/task~21833f8f.c9de4a56.js" rel="prefetch"><link href="/js/team-settings--cloud-hooks~21833f8f.7dc6865f.js" rel="prefetch"><link href="/js/team-settings--flow-concurrency~21833f8f.3b0a10d1.js" rel="prefetch"><link href="/js/team-settings--flow-groups~21833f8f.6345f181.js" rel="prefetch"><link href="/js/team-settings--kv~b07b7304.f618dbb1.js" rel="prefetch"><link href="/js/team-settings--members~21833f8f.017c3f6b.js" rel="prefetch"><link href="/js/team-settings--projects~21833f8f.b62b8227.js" rel="prefetch"><link href="/js/team-settings--secrets~b07b7304.44bf36dd.js" rel="prefetch"><link href="/js/team-settings--service-accounts~31ecd969.0ea6ce20.js" rel="prefetch"><link href="/js/team-settings--task-concurrency~21833f8f.3a66c27c.js" rel="prefetch"><link href="/js/team-settings--tokens~21833f8f.438eafcc.js" rel="prefetch"><link href="/js/team-settings~b07b7304.8865ec02.js" rel="prefetch"><link href="/js/team-switched~f1126255.40eea09c.js" rel="prefetch"><link href="/js/tutorials~3d9b8e9e.7dd7d50e.js" rel="prefetch"><link href="/js/user-settings--keys~712cfaab.3b19479b.js" rel="prefetch"><link href="/js/user-settings--profile~b07b7304.0df8af96.js" rel="prefetch"><link href="/js/user-settings--teams~21833f8f.d8d44012.js" rel="prefetch"><link href="/js/user-settings--tokens~21833f8f.814e8474.js" rel="prefetch"><link href="/js/user-settings~3bf82df3.8d0e2947.js" rel="prefetch"><link href="/css/app~d0ae3f07.91217051.css" rel="preload" as="style"><link href="/css/chunk-vendors~4a7e9e0b.e85830a2.css" rel="preload" as="style"><link href="/js/app~d0ae3f07.21daf4b4.js" rel="preload" as="script"><link href="/js/chunk-vendors~1f20a385.82b01cc1.js" rel="preload" as="script"><link href="/js/chunk-vendors~205977d4.9468c47c.js" rel="preload" as="script"><link href="/js/chunk-vendors~253ae210.28b20d05.js" rel="preload" as="script"><link href="/js/chunk-vendors~2ee72088.83b60546.js" rel="preload" as="script"><link href="/js/chunk-vendors~4a7e9e0b.806de680.js" rel="preload" as="script"><link href="/js/chunk-vendors~7274e1de.5de72067.js" rel="preload" as="script"><link href="/js/chunk-vendors~b58f7129.1b4555e8.js" rel="preload" as="script"><link href="/js/chunk-vendors~bd2a49b8.bff14397.js" rel="preload" as="script"><link href="/js/chunk-vendors~fdc6512a.23682883.js" rel="preload" as="script"><link href="/css/chunk-vendors~4a7e9e0b.e85830a2.css" rel="stylesheet"><link href="/css/app~d0ae3f07.91217051.css" rel="stylesheet"></head><body><noscript><strong>We're sorry but the Prefect UI doesn't work properly without JavaScript enabled. Please enable it to continue.</strong></noscript><div id="app"></div><div id="app-loader" class="loading"><span></span> <span></span> <span></span> <span></span> <span></span> <span></span> <span></span></div><script src="/js/chunk-vendors~253ae210.28b20d05.js"></script><script src="/js/chunk-vendors~7274e1de.5de72067.js"></script><script src="/js/chunk-vendors~1f20a385.82b01cc1.js"></script><script src="/js/chunk-vendors~2ee72088.83b60546.js"></script><script src="/js/chunk-vendors~b58f7129.1b4555e8.js"></script><script src="/js/chunk-vendors~fdc6512a.23682883.js"></script><script src="/js/chunk-vendors~bd2a49b8.bff14397.js"></script><script src="/js/chunk-vendors~205977d4.9468c47c.js"></script><script src="/js/chunk-vendors~4a7e9e0b.806de680.js"></script><script src="/js/app~d0ae3f07.21daf4b4.js"></script></body></html><style>#app-loader.loading {
  position: absolute;
  top: 50%;
  left: 50%;
  transform: translate(-50%, -50%);
}

#app-loader.loading span {
  display: inline-block;
  vertical-align: middle;
  width: 0.6em;
  height: 0.6em;
  margin: 0.19em;
  border-radius: 0.6em;
  animation: loading 1s infinite alternate;
}

#app-loader.loading span:nth-of-type(2) {
  background: #0082cb;
  animation-delay: 0.2s;
}
#app-loader.loading span:nth-of-type(3) {
  background: #27b1ff;
  animation-delay: 0.4s;
}
#app-loader.loading span:nth-of-type(4) {
  background: #0082cb;
  animation-delay: 0.6s;
}
#app-loader.loading span:nth-of-type(5) {
  background: #73e3ff;
  animation-delay: 0.8s;
}
#app-loader.loading span:nth-of-type(6) {
  background: #0082cb;
  animation-delay: 1s;
}
#app-loader.loading span:nth-of-type(7) {
  background: #27b1ff;
  animation-delay: 1.2s;
}

@keyframes loading {
  0% {
    opacity: 0;
  }
  100% {
    opacity: 1;
  }
}

Oct 25 '21 21:10 Adam-D-Lewis

It seems that prefect is set up to host prefect ui at a subdomain rather than a subpath. Not sure if we can get around this and still use a subpath. I tried setting the prefect.endpoint from https://github.com/PrefectHQ/prefect/blob/master/src/prefect/config.toml in the prefectConfig section of https://github.com/PrefectHQ/server/blob/master/helm/prefect-server/values.yaml, but it didn't seem to help.

This might be useful, https://community.traefik.io/t/serve-relative-paths-behind-traefik-pathprefix/2654

Alternatively, setting PREFECT_SERVER__BASE_URL to /prefect-ui/ may work.

Oct 26 '21 00:10 Adam-D-Lewis

For prefect we'll definitely need to set PREFECT_SERVER__BASE_URL. That's the only way for a we server to truly support a base url. With traefic we can rewrite urls but that won't handle everything.

Oct 26 '21 12:10 costrouc

After trying for a while unsuccesfully on Prefect, I've reached out to the Prefect team via a github issue to see if they have any feedback.

Oct 27 '21 16:10 Adam-D-Lewis

Hi @Adam-D-Lewis, any updates from the Prefect team on this?

Nov 24 '21 12:11 viniciusdc

From the development of the stages in PR #1003 I would like to propose a general "extension" mechanism that makes all parts of Qhub an extension. Take for example the following from deploy.py

def provision_01_terraform_state(stage_outputs, config):
     ...
     elif config['provider'] == 'gcp':
        stage_outputs[directory] = terraform.deploy(
            terraform_import=True,
            directory=os.path.join(directory, config['provider']),
            input_vars={
                'name': config['project_name'],
                'namespace': config['namespace'],
                'region': config['google_cloud_platform']['region']
            },
            terraform_objects=[
                QHubGCPProvider(config),
            ],
            state_imports=[(
                "module.terraform-state.module.gcs.google_storage_bucket.static-site",
                f"{config['project_name']}-{config['namespace']}-terraform-state",
            )])
    ...

Later when we look at stages/02-infrastructure/gcp we see a terraform module /directory that is called by this function. There is a lot being done in the terraform.deploy(...) function but there are important sections seen here.

input_vars how do values in qhub_config.yaml and outputs from prior stages stage_outputs map to imput variables to the terraform module.
state_imports these are ids/attrs that can be used such that terraform can "import" these resources if they already exist. This allows for qhub to be more stateless. We have used this in the past to import terraform-state in the stages/01-terraform-state.
terraform_objects is a hook to allow for "dynamic" terraform resources to be rendered to the stage. This should only be a function of the qhub_config.yaml and NOT previous stage_outputs so that the render command can be separated from the deploy command.

Right now this imformation is written redundantly in qhub/deploy.py and qhub/destroy.py instead I suggest that this information be included in the terraform module itself and qhub will search for it. My suggestion is config.py.

The structure I am suggesting. A minimal QHub extension would be as follows.

config.py
*.tf

config.py would be an optional file and if nothing is provided we would assume sane defaults:

input_vars would be set to {"qhub_config": config, "stage_outputs": state_outputs}
state_imports would be empty []
terraform_outputs would be empty []

The optional functions within config.py would be

def input_vars(qhub_config, stage_outputs):
     return {
         'name': qhub_config['project_name']
    }

def state_imports(qhub_config, stage_outputs):
     return [(
                "module.terraform-state.module.spaces.digitalocean_spaces_bucket.main",
                f"{qhub_config['digital_ocean']['region']},{qhub_config['project_name']}-{qhub_config['namespace']}-terraform-state",
    )]

def terraform_ouputs(qhub_config):
    from qhub.render.terraform import QHubGCPProvider
    return [
           QHubGCPProvider(config),
    ]

terraform_outputs would be called within the render stage. While state_imports and input_vars would be called in the deploy stage. The advantage here is that we could fully adopt this model for all of our components making us fully use our extension mechanism within qhub.

Questions

How would you allow additional extensions easily in QHub with this model?

extensions:
   - path: github.com/path/to/repo
     config:
        a: "value"
   - path: ./my/custom/qhub/extension
     config:
         another: 1
         value: [1, 2, 3]

We would add an extensions key to qhub-config.yaml that takes a list of paths. These paths can be directories and github repo paths as well. After the core qhub stages have been run it will iterate through each extension using the config.py in the extension to determine the mapping of qhub-config.yaml to terraform input variables. Extensions can also assume that certain providers are already configured via environment variables e.g. keycloak and kubernetes.

The main reason I suggest this approach is that it allows for easy abstraction and making qhub-enterprise just another extension to qhub.

Jan 31 '22 20:01 costrouc

I also would like to suggest another incremental option to qhub deployment, following the same idea as above. I think it might be interesting (for testing and CI purposes) to enable the qhub deployment to be split into phases, e.g qhub deploy -c qhub-config.yaml --stage 02-infrastructure or qhub deploy -c qhub-config.yaml --stages [02-infrastructure, 03-...]

This would allow qhub to be deployed from any stage, e.g. if an error occurs during deployment we can target that stage onward post fix...

But the great benefit in my opinion, would be within CI testing, as we could have:

runs:
  using: ...
  steps:

  - name: Qhub 01-terraform-state provision
    shell: bash
    run: |
      qhub deploy -c config.yaml --stages 01-terraform-state

 (Do something in between, like checking if state group exists..)
 
 - name: Qhub 02-infraestructure provision
    shell: bash
    run: |
      qhub deploy -c config.yaml --stages 02-...

(kubectl to check current resources, test LB..., inspect vars...)

The benefit in doing so would be that we could:

use those rendered configs for each stage to pass a simple .tf linting,
check any python code for that stage
inspect if the .json output matches a given template (?)
We will have the ability to perform outside (from qhub and terraform) tests with the current deployed infra

Feb 25 '22 20:02 viniciusdc

Added an RFD https://github.com/nebari-dev/governance/issues/35

Mar 28 '23 13:03 costrouc

nebari nebari copied to clipboard

QHub Extension Mechanism

Description

Suggestion

OAuth2 Provisioning

Ingress Provisioning

Monitoring Integration

Questions

nebari
nebari copied to clipboard