SDK for building and deploying custom sources

[chrismatix]

Is your feature request related to a problem? Please describe.

To deploy custom user code to the Airy cluster we need to integrate the runtimes with our existing Kubernetes setup.

Describe the solution you'd like

Sources are installed as Helm charts. There is a built-in chart that can be installed by running a command like so:

helm install source-facebook infrastructure/charts/components/sources/custom \ 
--source.imageTag=ghcr.io/airyhq/source-facebook:latest \
--set source.name=facebook \ 
--set. sources.secrets.sourceToken=token \ 
--set sources.secrets.authToken=123 \ 
--set sources.secrets.appId=123

Where the custom directory contains a template helm chart. The name and the image tag are required. Optionally source users can provide necessary secrets using the source.secrets map, which will be included as a secret to the deployment.

• [ ] Docs
• [ ] Add default deployment
• [ ] Add default ingress controller
• [ ] Auto create source.events topic in case the source wants to use an ingestion buffer

[chrismatix]

@ljupcovangelski What do you think about foregoing all this config and packaging and installing custom sources via Helm?

[ljupcovangelski]

We can make every custom source it's own independent helm chart. That way it can also be part of the umbrella chart and deployed also separately.

In that sense, every custom source chart should bring its own resources, including ingress routes, secrets, configMaps.

I expect the installation to be something like:

helm install sourde-viber    infrastructure/charts/components/sources/custom --set name=viber --set credentials=...
helm install sourde-telegram infrastructure/charts/components/sources/custom --set name=telegram --set credentials=...

Eventually I don't see why we shouldn't make it part of the airy.yaml file and part of airy config apply and airy upgrade. So in my opinion the definition should look something like this:

components:
  sources:
    facebook:
      ...
      ...
    custom:
      - name: viber
        credentials:
          - viberid: ...
          - viberpass: ...
      - name: telegram
        credentials:
          - telegramkey: ...
          - telegramsafile: ...