Support external config schemas in the backend

[davidfestal]

Hey, I just made a Pull Request!

On the backend application, thisPR allows referencing additional configuration schemas at runtime (application start), and have them taken in account where schemas are used:

• the backed-app plugin, in order to prepare the frontend configuration
• the backend application loggers, in oder to hide secret configuration values (defined in the schemas)

This fixes #21631

:heavy_check_mark: Checklist

• [x] A changeset describing the change and affected packages. (more info)
• [ ] Added or updated documentation
• [x] Tests for new functionality and regression tests for bug fixes
• [ ] Screenshots attached (for UI changes)
• [x] All your commits have a Signed-off-by line in the message. (more info)

[backstage-goalie[bot]]

Changed Packages

Package Name	Package Path	Changeset Bump	Current Version
@backstage/backend-app-api	packages/backend-app-api	patch	v0.5.11-next.2
@backstage/backend-dynamic-feature-service	packages/backend-dynamic-feature-service	minor	v0.1.1-next.2
@backstage/plugin-app-backend	plugins/app-backend	patch	v0.3.58-next.2
@backstage/plugin-app-node	plugins/app-node	patch	v0.1.10-next.2

[github-actions[bot]]

Uffizzi Cluster pr-21632 was deleted.

[github-actions[bot]]

This PR has been automatically marked as stale because it has not had recent activity from the author. It will be closed if no further activity occurs. If the PR was closed and you want it re-opened, let us know and we'll re-open the PR so that you can continue the contribution!

[davidfestal]

@Rugvip Could this PR be reopened ? Thanks !

[freben]

Sure!

[github-actions[bot]]

This PR has been automatically marked as stale because it has not had recent activity from the author. It will be closed if no further activity occurs. If the PR was closed and you want it re-opened, let us know and we'll re-open the PR so that you can continue the contribution!

[davidfestal]

@Rugvip @freben Could this PR be reopened ? Thanks ! Any idea whether it could get a review ?

[davidfestal]

@Rugvip @backstage-service Are these errors in the e2e tests known: https://github.com/backstage/backstage/actions/runs/7463481178/job/20308264561?pr=21632#step:11:143

Or do they ring a bell to anyone ?

This seems to be something frontend-related, and I don't see how it would have anyting to do with the changes done in this PR (at least afaict). Any help would be appreciated.

[davidfestal]

@Rugvip This has been rebased and completed.

[backstage-goalie[bot]]

Thanks for the contribution! All commits need to be DCO signed before they are reviewed. Please refer to the the DCO section in CONTRIBUTING.md or the DCO status for more info.

[davidfestal]

@Rugvip Is the PR now OK for you, or do you want some other option to be implemented for comment https://github.com/backstage/backstage/pull/21632#discussion_r1461001122 ?

[github-actions[bot]]

This PR has been automatically marked as stale because it has not had recent activity from the author. It will be closed if no further activity occurs. If the PR was closed and you want it re-opened, let us know and we'll re-open the PR so that you can continue the contribution!

[davidfestal]

@Rugvip Is the PR now OK for you, or do you want some other option to be implemented for comment #21632 (comment) ?

@Rugvip Could you please remove the stale label ? It seems to me that this PR was quite near to be merged, though I was still waiting for answer to my question above. This PR is quite important to have the dynamic pkugins be really usable in upstream backstage. Do you think you could provide me with your thoughts about what you expect me to further fix to have it merged ?

[davidfestal]

@Rugvip Please see comment https://github.com/backstage/backstage/pull/21632#discussion_r1480368731

It seems that now all your comments / requests for change have been fixed. I have also rebased the PR on top of master.

Is it OK to merge it now ?

cc @christophe-f

[davidfestal]

@Rugvip Please see comment #21632 (comment)

It seems that now all your comments / requests for change have been fixed. I have also rebased the PR on top of master.

Is it OK to merge it now ?

cc @christophe-f

In the following comment I added a more complete answer to your question about lifting up the API one level or two : https://github.com/backstage/backstage/pull/21632#discussion_r1482778993

Hopefully it would help.

[davidfestal]

@Rugvip I applied your proposal in commit https://github.com/backstage/backstage/pull/21632/commits/2fb527dd34dab57917005d6e0b51a1c5332d4f99 so that:

• there is no more alpha schemaDiscovery service
• minimal changes to the API boil down to providing an option to pass an optional ConfigSchema to either the createConfigSecretEnumerator or the app-backend plugin (through an extension)
• Dynamic plugin schema support is added by an alternate rootLogger service implementation and an app-backend plugin module available in the backend-dynamic-features-service.

Hopefully this design would make this PR mergeable ?

[davidfestal]

@Rugvip I applied your proposal in commit 2fb527d so that:

• there is no more alpha schemaDiscovery service
• minimal changes to the API boil down to providing an option to pass an optional ConfigSchema to either the createConfigSecretEnumerator or the app-backend plugin (through an extension)
• Dynamic plugin schema support is added by an alternate rootLogger service implementation and an app-backend plugin module available in the backend-dynamic-features-service.

Hopefully this design would make this PR mergeable ?

However the verify check failed due to this error:

Plugin package '@backstage/backend-dynamic-feature-service' with role 'node-library' has a runtime dependency on package '@backstage/backend-app-api', which is not permitted. If you are using this dependency for dev server purposes, you can move it to devDependencies instead.

Should the createConfigSecretEnumerator() function be moved to another package ?

For now I added this as a legitimate exception in https://github.com/backstage/backstage/pull/21632/commits/b0f33dc77a33c0e6187c9dae594cdacf444df6cb#diff-60885cb9275cb61d02f9f11f56ecc3a8a4014477cf6dfc5ad8e6788946ab3e44R93

[Rugvip]

Should the createConfigSecretEnumerator() function be moved to another package ?

Yep we've been discussing this a bit, and we may move things out of backend-app-api. Current solution is good for now 👍

[github-actions[bot]]

Thank you for contributing to Backstage! The changes in this pull request will be part of the 1.23.0 release, scheduled for Tue, 20 Feb 2024.