RBAC Role Data Out-of-Sync When Scaling Horizontally

[zthfmh]

Describe the bug

Hello, we've discovered some inconsistent behavior from the rbac-backend plugin when scaling the backstage deployment to 2 pods/containers with identical instance. The use case we had issue with is when using the API endpoint PUT </api/permission/roles/:kind/:namespace/:name> (https://github.com/janus-idp/backstage-plugins/blob/main/plugins/rbac-backend/docs/apis.md#put-role) to update membership of a role. When having two Kubernetes pods running, the API request was received and processed by pod A, in which itself also updated the DB with the new role definition. Everything seemed good at this point until someone started querying the endpoint GET </api/permission/roles/:kind/:namespace/:name> (https://github.com/janus-idp/backstage-plugins/blob/main/plugins/rbac-backend/docs/apis.md#get-role) to retrieve the role - the updated role data was returned when a request hit pod A, however if the same request was received by pod B, we'd still get the old role definition.

Since the DB appears to be successfully updated, we are wondering if the plugin utilizes some kind of in memory cache that caused this out-of-sync behavior. Our backstage deployment has been configured to using Redis as the cache backend and it's seemingly not taking advantage of that. Does anyone have any thoughts on this? Thanks in advance.

Expected Behavior

When having a high availability deployment setup, API endpoints served by the RBAC backend plugin should provide consistent data across all instances.

What are the steps to reproduce this bug?

1. Horizontally scale the backstage deployment to have 2 replicas
2. Choose any existing RBAC role, and retrieve its current definition with the endpoint GET </api/permission/roles/:kind/:namespace/:name>
3. Modify the retrieved definition by adding a new member to the role. Feed the endpoint PUT </api/permission/roles/:kind/:namespace/:name> with the new definition to update the role
4. Querying GET </api/permission/roles/:kind/:namespace/:name>, you will see that the data returned varies depending on the pod/container being hit

Versions of software used and environment

@backstage/app-defaults 1.5.3 @backstage/backend-app-api 0.5.14, 0.6.2 @backstage/backend-common 0.19.9, 0.20.1, 0.21.6 @backstage/backend-dev-utils 0.1.4 @backstage/backend-openapi-utils 0.1.9 @backstage/backend-plugin-api 0.6.16 @backstage/backend-tasks 0.5.21 @backstage/backend-test-utils 0.3.6 @backstage/catalog-client 1.6.3 @backstage/catalog-model 1.4.5 @backstage/cli 0.26.2 @backstage/cli-common 0.1.13 @backstage/cli-node 0.2.4 @backstage/config 1.2.0 @backstage/config-loader 1.7.0 @backstage/core-app-api 1.12.3 @backstage/core-compat-api 0.2.3 @backstage/core-components 0.12.5, 0.13.10, 0.14.3 @backstage/core-plugin-api 1.9.1 @backstage/dev-utils 1.0.30 @backstage/e2e-test-utils 0.1.1 @backstage/errors 1.2.4 @backstage/eslint-plugin 0.1.6 @backstage/frontend-plugin-api 0.6.3 @backstage/integration 1.9.1 @backstage/integration-aws-node 0.1.12 @backstage/integration-react 1.1.25 @backstage/plugin-adr 0.6.16 @backstage/plugin-adr-backend 0.4.13 @backstage/plugin-adr-common 0.2.21 @backstage/plugin-api-docs 0.11.3 @backstage/plugin-app-backend 0.3.64 @backstage/plugin-app-node 0.1.16 @backstage/plugin-auth-backend 0.22.3 @backstage/plugin-auth-backend-module-atlassian-provider 0.1.8 @backstage/plugin-auth-backend-module-aws-alb-provider 0.1.8 @backstage/plugin-auth-backend-module-gcp-iap-provider 0.2.11 @backstage/plugin-auth-backend-module-github-provider 0.1.13 @backstage/plugin-auth-backend-module-gitlab-provider 0.1.13 @backstage/plugin-auth-backend-module-google-provider 0.1.13 @backstage/plugin-auth-backend-module-microsoft-provider 0.1.11 @backstage/plugin-auth-backend-module-oauth2-provider 0.1.13 @backstage/plugin-auth-backend-module-oauth2-proxy-provider 0.1.9 @backstage/plugin-auth-backend-module-oidc-provider 0.1.7 @backstage/plugin-auth-backend-module-okta-provider 0.0.9 @backstage/plugin-auth-node 0.4.11 @backstage/plugin-auth-react 0.0.3 @backstage/plugin-catalog 1.18.2 @backstage/plugin-catalog-backend 1.21.0 @backstage/plugin-catalog-backend-module-aws 0.3.11 @backstage/plugin-catalog-backend-module-gitlab 0.3.14 @backstage/plugin-catalog-backend-module-scaffolder-entity-model 0.1.14 @backstage/plugin-catalog-backend-module-unprocessed 0.4.3 @backstage/plugin-catalog-common 1.0.22 @backstage/plugin-catalog-graph 0.4.3 @backstage/plugin-catalog-import 0.10.9 @backstage/plugin-catalog-node 1.11.0 @backstage/plugin-catalog-react 1.11.2 @backstage/plugin-catalog-unprocessed-entities 0.2.2 @backstage/plugin-catalog-unprocessed-entities-common 0.0.1 @backstage/plugin-devtools 0.1.12 @backstage/plugin-devtools-backend 0.3.2 @backstage/plugin-devtools-common 0.1.9 @backstage/plugin-events-node 0.2.21, 0.3.2 @backstage/plugin-github-actions 0.6.14 @backstage/plugin-home 0.7.2 @backstage/plugin-home-react 0.1.11 @backstage/plugin-kubernetes 0.11.8 @backstage/plugin-kubernetes-backend 0.16.4 @backstage/plugin-kubernetes-common 0.7.5 @backstage/plugin-kubernetes-node 0.1.10 @backstage/plugin-kubernetes-react 0.3.3 @backstage/plugin-org 0.6.23 @backstage/plugin-pagerduty 0.7.5 @backstage/plugin-permission-backend 0.5.40 @backstage/plugin-permission-common 0.7.13 @backstage/plugin-permission-node 0.7.27 @backstage/plugin-permission-react 0.4.21 @backstage/plugin-proxy-backend 0.4.14 @backstage/plugin-scaffolder 1.19.2 @backstage/plugin-scaffolder-backend 1.22.3 @backstage/plugin-scaffolder-backend-module-azure 0.1.8 @backstage/plugin-scaffolder-backend-module-bitbucket 0.2.6 @backstage/plugin-scaffolder-backend-module-bitbucket-cloud 0.1.6 @backstage/plugin-scaffolder-backend-module-bitbucket-server 0.1.6 @backstage/plugin-scaffolder-backend-module-cookiecutter 0.2.40 @backstage/plugin-scaffolder-backend-module-gerrit 0.1.8 @backstage/plugin-scaffolder-backend-module-gitea 0.1.6 @backstage/plugin-scaffolder-backend-module-github 0.2.6 @backstage/plugin-scaffolder-backend-module-gitlab 0.3.2 @backstage/plugin-scaffolder-common 1.5.1 @backstage/plugin-scaffolder-node 0.2.9, 0.4.2 @backstage/plugin-scaffolder-react 1.8.3 @backstage/plugin-search 1.4.9 @backstage/plugin-search-backend 1.5.6 @backstage/plugin-search-backend-module-catalog 0.1.21 @backstage/plugin-search-backend-module-elasticsearch 1.3.19 @backstage/plugin-search-backend-module-pg 0.5.25 @backstage/plugin-search-backend-module-techdocs 0.1.21 @backstage/plugin-search-backend-node 1.2.20 @backstage/plugin-search-common 1.2.11 @backstage/plugin-search-react 1.7.9 @backstage/plugin-shortcuts 0.3.22 @backstage/plugin-signals-node 0.0.1 @backstage/plugin-signals-react 0.0.1 @backstage/plugin-sonarqube 0.7.15 @backstage/plugin-sonarqube-backend 0.2.18 @backstage/plugin-sonarqube-react 0.1.14 @backstage/plugin-tech-radar 0.7.2 @backstage/plugin-techdocs 1.10.3 @backstage/plugin-techdocs-backend 1.10.3 @backstage/plugin-techdocs-module-addons-contrib 1.1.8 @backstage/plugin-techdocs-node 1.12.2 @backstage/plugin-techdocs-react 1.2.2 @backstage/plugin-user-settings 0.8.4 @backstage/release-manifests 0.0.11 @backstage/test-utils 1.5.3 @backstage/theme 0.2.19, 0.4.4, 0.5.2 @backstage/types 1.1.1 @backstage/version-bridge 1.0.7 @janus-idp/backstage-plugin-feedback 1.0.1 @janus-idp/backstage-plugin-feedback-backend 1.1.1 @janus-idp/backstage-plugin-rbac 1.13.2 @janus-idp/backstage-plugin-rbac-backend 2.5.0 @janus-idp/backstage-plugin-rbac-common 1.2.1, 1.3.1 @janus-idp/backstage-plugin-rbac-node 1.0.4 @k-phoen/backstage-plugin-confluence 0.0.15 @k-phoen/backstage-plugin-confluence-backend 0.0.14 @k-phoen/backstage-plugin-grafana 0.1.22 @roadiehq/backstage-plugin-argo-cd 2.5.1 @roadiehq/backstage-plugin-datadog 2.2.2 @roadiehq/backstage-plugin-home-rss 1.3.5