feat: setup configuration-ui plugin registry `Draft`

[e2corporation]

⚠️ Config-UI / Plugin Registry

WIP DO NOT MERGE

• [ ] Create a Defined Plugin Registration Schema (JSON Registration Files)
• [ ] Create Local Plugin Registry Configuration
  • [ ] JIRA
  • [ ] GitLab
  • [ ] GitHub
  • [ ] Jenkins
  • [ ] TAPD
  • [ ] AE
  • [ ] DBT
  • [ ] StarRocks
  • [ ] RefDiff
  • [ ] GitExtractor
  • [ ] Feishu
• [ ] Implement useIntegrations Hook
  • [ ] Create Plugin and other related Data Models
  • [ ] Define Integrations Data Configuration Objects
  • [ ] Validate JSON Plugin Configuration
• [ ] Create Dynamic / Live Plugin Registry (Allow Plugins to load via API)
• [ ] Create Basic Documentation for Plugin JSON Registration
• [ ] Refactor Integrations Data Dependencies

Description

This PR creates a Plugin Registry on Configuration-UI as outlined on #2882.

Milestone

TBD

Local Plugin Registry

The local registry defines plugins that are bundled with DevLake. Users creating "Private" local plugins may add and register plugins with the local registry.

API Plugin Registry

The api registry allows plugins to be defined by the backend and plugins can be fetched dynamically and merged with the local registry.

Plugin Registration Schema (JSON)

{
  "id": "gitlab",
  "type": "integration",
  "enabled": true,
  "multiConnection": true,
  "isBeta": false,
  "isProvider": true,
  "name": "GitLab",
  "icon": "src/images/integrations/gitlab.svg",
  "private": false,
  "connection": {
    "authentication": "token",
    "fields": {
      "name": { "enable": true, "required": true, "readonly": false },
      "endpoint": { },
      "proxy": { },
      "token": { },
      "rateLimitPerHour": { }
    },
    "labels": {
      "name": "Connection Name",
      "endpoint": "Endpoint URL",
      "proxy": "Proxy URL",
      "token": "Access Token",
      "rateLimitPerHour": "Rate Limit (per hour)"
    },
    "placeholders": {
      "name": "eg. GitLab",
      "endpoint": "eg. https://gitlab.com/api/v4/",
      "proxy": "eg. http://proxy.localhost:8080",
      "token": "eg. ff9d1ad0e5c04f1f98fa",
      "rateLimitPerHour": "1000"
    },
    "tooltips": {}
  },
  "entities": ["CODE", "TICKET", "CROSS"]
}

Does this close any open issues?

#2882

Screenshots

[<PENDING>]

[tk103331]

Wouldn't it be better to define the properties of the field like this:

...
{
    "authentication": "token",
    "fields": [
        {
            "name": "name",
            "label": "Connection Name",
            "placeholder": "eg.GitLab",
            "enable": true,
            "required": true,
            "readonly": false
        },
        {
            "name": "endpoint",
            "label": "Endpoint URL",
            "placeholder": "eg. https://gitlab.com/api/v4/"
        },
        {
            "name": "token",
            "label": "Access Token",
            "placeholder": "eg. ff9d1ad0e5c04f1f98fa"
        }
    ]
}
...

This has at least the following benefits:

• Focus on all properties of a field when defining fields, rather than defining them in scattered places.
• The form of the fileds array can solve the sorting problem of the fields, and the way of using the JSON object is likely to cause the problem of the traversal order.

I do see your point, I wouldn't say it's scattered, sorting of the fields is not something needed or relevant, the order of the fields in the connection form follows a natural order starting with Name, ending with Rate Limit.

In addition, do you need to add some other attributes to the field, such as

• showType, which represents front-end display components, such as input, select, checkbox, etc.
• validation, which represents standard validation rules, including regular validation, length validation, etc.
• multiple, which means that it is a multiple format, such as the PAT of the GitHub plugin, you can fill in multiple.

I see the point, and we can certainly evolve the configuration to let validation rules be defined here but it's not needed right now. Instead of showType we would just use type (string|number|object) etc and add a component prop to specify a form control that's to be used for render.

[likyh]

Wouldn't it be better to define the properties of the field like this:

...
{
    "authentication": "token",
    "fields": [
        {
            "name": "name",
            "label": "Connection Name",
            "placeholder": "eg.GitLab",
            "enable": true,
            "required": true,
            "readonly": false
        },
        {
            "name": "endpoint",
            "label": "Endpoint URL",
            "placeholder": "eg. https://gitlab.com/api/v4/"
        },
        {
            "name": "token",
            "label": "Access Token",
            "placeholder": "eg. ff9d1ad0e5c04f1f98fa"
        }
    ]
}
...

This has at least the following benefits:

• Focus on all properties of a field when defining fields, rather than defining them in scattered places.
• The form of the fileds array can solve the sorting problem of the fields, and the way of using the JSON object is likely to cause the problem of the traversal order.

In addition, do you need to add some other attributes to the field, such as

• showType, which represents front-end display components, such as input, select, checkbox, etc.
• validation, which represents standard validation rules, including regular validation, length validation, etc.
• multiple, which means that it is a multiple format, such as the PAT of the GitHub plugin, you can fill in multiple.

There is some information in this issue #2846

I think so.

[likyh]

No matter how to do it, it looks like we must use one config to control all the logic. I think #2862 「delete ProviderIcons and ProviderFormPlaceholders and replace them with one config ProviderConfigMap. (integrationsData will refactor in future)」 and 「replace some switch(provider.id)/[xxx,yyy].includes(provider.id)/[jira,github] with more common code.」 are necessary. Why you hate it so much?

[e2corporation]

No matter how to do it, it looks like we must use one config to control all the logic. I think #2862 「delete ProviderIcons and ProviderFormPlaceholders and replace them with one config ProviderConfigMap. (integrationsData will refactor in future)」 and 「replace some switch(provider.id)/[xxx,yyy].includes(provider.id)/[jira,github] with more common code.」 are necessary. Why you hate it so much?

You go through all the trouble of introducing another foreign ProviderConfigMap which is a hack workaround to stuff all the needed props into one place, add a note that you have to refactor it more in the future -- and you think that's a better solution than the well defined JSON config files? Your solution was more out of desperation than a proper way to evolve the configuration system. I only have 1 commit on this PR -- there are more changes to be added. I already mentioned that the switch statements in connection manager will be consolidated and removed.

[e2corporation]

Wouldn't it be better to define the properties of the field like this:

...
{
    "authentication": "token",
    "fields": [
        {
            "name": "name",
            "label": "Connection Name",
            "placeholder": "eg.GitLab",
            "enable": true,
            "required": true,
            "readonly": false
        },
        {
            "name": "endpoint",
            "label": "Endpoint URL",
            "placeholder": "eg. https://gitlab.com/api/v4/"
        },
        {
            "name": "token",
            "label": "Access Token",
            "placeholder": "eg. ff9d1ad0e5c04f1f98fa"
        }
    ]
}
...

This has at least the following benefits:

• Focus on all properties of a field when defining fields, rather than defining them in scattered places.
• The form of the fileds array can solve the sorting problem of the fields, and the way of using the JSON object is likely to cause the problem of the traversal order.

In addition, do you need to add some other attributes to the field, such as

• showType, which represents front-end display components, such as input, select, checkbox, etc.
• validation, which represents standard validation rules, including regular validation, length validation, etc.
• multiple, which means that it is a multiple format, such as the PAT of the GitHub plugin, you can fill in multiple.

There is some information in this issue #2846

I thought about this as well, and I agree to a certain extent and may re-configure the json files to be this way. The JSON schema is still being evaluated and worked on. For now it's easier on the user to edit 3 separate objects than 1 config object with 2nd level keys for everything. Additionally, it let's me construct the existing Variable Configurations without having to reduce the object configuration and transform it again into the structure I need.

[e2corporation]

A Plugin does not need to be responsible for determining it's own Connection Fields at this time with regards to them being dynamically created, it should be following a standard convention so Each Plugin has to satisfy the current Connection model defined by the backend. The plugin can have dynamic control (enabled state, required state etc) over the standard connection fields that exist currently, which is what this configuration is intended for. Based on a plugin's authentication type that also is a main factor as to what main fields are required for the connection.

In the future the plugin can perhaps have more control and custom rendering options for the field. This is something to be discussed in more detail.

[likyh]

It's the first PR and ProviderConfigMap can be replaced with JSON or API requests at any time.

In addition, my two PRS may not be merged. But please don't create PR so big. It's an open-source project.

[e2corporation]

It's the first PR and ProviderConfigMap can be replaced with JSON or API requests at any time.

In addition, my two PRS may not be merged. But please don't create PR so big. It's an open-source project.

ProviderConfigMap is short term hack. Had you invested time in discussing with me what your intentions were you could have saved yourself a lot of time, Instead you choose to refactor on your own and make your own decisions, and create a PR with changes that go against my design strategy.

[e2corporation]

@likyh @tk103331 I'll be making more commits as I continue ideating on the Plugin Registry concept. In the mean time, I'm fine with both of you contributing thoughts & ideas to this effort, that are hopefully inline with this strategy.

@tk103331 Feel free to comment on more suggestions/questions or issues for the Plugin Schema as it's being developed, some option configurations can be for a future need as well. The important thing is to get an essential schema for the short-term without overcomplicating it. You will also be able to add commits if you are wanting to contribute.

@likyh When I'm done with a few more commits, I'll tag you to adapt some of your valid refactor changes from your closed PR, such as the Dynamic Menu from new $Integrations var. There are also areas of the useIntegrations hook you can extend, such as the validation handler to verify a plugin's object properties meets Schema specifications, as well as working on live API registry.

[tk103331]

@e2corporation Sorry, I didn't see what you edited in my reply two days ago.

I do see your point, I wouldn't say it's scattered, sorting of the fields is not something needed or relevant, the order of the fields in the connection form follows a natural order starting with Name, ending with Rate Limit.

What I mean is to spread the multiple information (name, label, placeholder) of a field to be defined in multiple places instead of using only one object. Regarding field sorting, in general, we will use the code writing order. However, if fileds is data requested from the backend. Possibly, the backend may use the map data structure (because the field is not fixed and cannot use struct), which will cause problems due to the uncertainty of the map traversal order. Of course, we can provide some instructions or helpers to avoid this problem, but it really depends on how the backend people write the code.

I see the point, and we can certainly evolve the configuration to let validation rules be defined here but it's not needed right now. Instead of showType we would just use type (string|number|object) etc and add a component prop to specify a form control that's to be used for render.

Yes, the name component would be more appropriate, the name showType is a premature suggestion.

[likyh]

Is this PR will continue now?

[e2corporation]

Is this PR will continue now?

I will be refreshing this soon, ideally other refactor PRs need to be completed or halted while I get this initial version of dynamic config ready.

[e2corporation]

@klesh @likyh @mintsweet Phase 1 Of Dynamic Plugin Registry is ready for main branch next so more development & refactor can continue. Thanks.

[e2corporation]

@likyh @mintsweet All notable code review feedback items have been resolved, any additional cleanups will be deferred to a subsequent PR.