Handle reduction of schemas in steve schema requests

[nwmac]

SURE-7040, SURE-6584

https://github.com/rancher/rancher/issues/38468

Backend would like to store less schemas - but this breaks YAML edit.

Need to investigate further

Update: The new backend issue is https://github.com/rancher/rancher/issues/41301 and is targeted for 2.7 Q3. We need to sync merging changes to all repos together

[richard-cox]

Tagging some relevent issues/prs

• https://github.com/rancher/rancher/issues/38468#issuecomment-1289463085
• https://github.com/rancher/steve/pull/62
• https://github.com/rancher/rancher/issues/39568
• https://github.com/rancher/rancher/issues/39567

[richard-cox]

~I thought we had a test container but couldn't find a reference.~ See slack

From my understanding the proposed change are

• Only primary schemas are returned from /v1/schemas (not sure how we define primary, but those that aren't solely part of resourceFields entries)
• Schemas are only omitted from /v1/schemas, the schemas themselves remain unchanged

From my understanding the UI needs

• The endpoint to fetch base schemas to remain as /v1/schemas
• A way to fetch schemas that are now not included in /v1/schemas

Questions

• Do the changes also apply to /k8s/clusters/<cluster id>/v1/schemas? In theory this shouldn't make any difference, just increases the impact
• To confirm, the changes to not affect v3 / Norman?
• Can we still make /v1/schemas requests to specific schemas that aren't included in /v1/schemas?

References to resourceFields

• const receiverSchema = this.$store.getters'cluster/schemaFor'; (monitoring.coreos.com.v1alpha1.alertmanagerconfig.spec.receivers)
• const routeSchema = this.$store.getters'cluster/schemaFor'; ('monitoring.coreos.com.v1alpha1.alertmanagerconfig.spec.route)
• const specSchema = this.$store.getters'cluster/schemaFor'; (monitoring.coreos.com.receiver.spec)
• const specSchema = this.$store.getters['cluster/schemaFor'](get(this.schema, 'resourceFields.spec.type') || ''); (cis.cattle.io.clusterscan)
• this.$getters['schemaFor'](get(schema, 'resourceFields.spec.type') || ''); (cis.cattle.io.clusterscan)
• pathExistsInSchema iterates through spec references (getting their schemas)
  • this.$store.getters${ inStore }/pathExistsInSchema;
  • this.$rootGetters'cluster/pathExistsInSchema';
• create-yaml.js

I've also reviewed general schemaFor and byId'](SCHEMA references and there was nothing of note

Given the above (including questions and assumptions), we need to confirm if we can change the getters to find (where we make http calls to specific schemas).

Update: We also use a schema's resourceFeilds whenever a form (not yaml!) saves

• resource-class validationErrors const fields = schema.resourceFields || {};

[richard-cox]

After chatting with Neil I think I got one of the assumptions wrong. Missing schema's will not be available via steve, we would need to go out directly fetch their openapi counterpart(?). We'd need time to work out if this is possible on top of the impact from comment above. I'm not sure we have time in the 2.7next1 timeframe

[gaktive]

@catherineluse to investigate this, though for @samjustus' benefit, UI is pushing this out to the Q2 release so the related backend work should not be merged.

[gaktive]

We need to resync with backend on their latest as we have many questions.

[richard-cox]

@gaktive I found some inline questions , the big one is described here... how do we make API requests to fetch schemas now not returned by steve, can they provide some concrete examples that work today?

[samjustus]

@richard-cox who is they? team1?

[richard-cox]

@samjustus They being the bucket term for backend. Not sure on teams, but @MbolotSuse was the owner of the PR for for the BE work to remove schemas and is probably best positioned to respond. We should discuss this in the next weekly performance call

[MbolotSuse]

Answers to questions:

Do the changes also apply to /k8s/clusters//v1/schemas? In theory this shouldn't make any difference, just increases the impact

That's correct, AFAIK they are the same endpoint (you may not see this in the test image since I didn't build a corresponding agent image) - one just runs on the stuff in local (/v1/schemas) and one runs on stuff in the downstream (/k8s/clusters/<cluster_id>/v1/schemas).

To confirm, the changes to not affect v3 / Norman?

Correct.

Can we still make /v1/schemas requests to specific schemas that aren't included in /v1/schemas?

Not in the current implementation. That endpoint basically returns a large map of all of the schemas available. Making a request to a specific endpoint just give you the schema for that resource, same as you see in the map (meaning that the pods entry that you see in /v1/schemas is the same as the entry for /v1/schemas/pods).

After chatting with Neil I think I got one of the assumptions wrong. Missing schema's will not be available via steve, we would need to go out directly fetch their openapi counterpart(?). We'd need time to work out if this is possible on top of the impact from comment above. I'm not sure we have time in the 2.7next1 timeframe

This part is correct. We would be entirely removing a large number of current schemas - but these represent field definitions (i.e. PodSpec), not top-level resources.

Some clarifications that may help:

Only primary schemas are returned from /v1/schemas (not sure how we define primary, but those that aren't solely part of resourceFields entries)

Basically, we only return the top-level fields and don't recursively redefine structs. For example, say I have a resource TestResource that has one field Spec TestSpec. We will expose a schema for TestResource, but we won't define the fields in TestSpec - you'll know the type exists but won't know about any of the fields there.

Schemas are only omitted from /v1/schemas, the schemas themselves remain unchanged

This is correct. You can still ask k8s for the schemas if you want the full definition, but that operation seems to be exceedingly slow.

FWIW, one approach that we discussed was exposing a /v1/resources endpoint which functions as the above does (top-level only fields or no fields at all) and keeping around the /v1/schemas fields. The UI would then pull in /v1/resources frequently (for checks on collectionVerbs and the like - button/element visibility would be addressed by this endpoint), but only infrequently target /v1/schemas for specific resources that you need the full definition of. Though that's not implemented at all in the current state.

[richard-cox]

Thanks Michael.

Would it be possible to add some examples, given the features in the current PR, of fetching schemas that have been dropped? So for instance how would we get the PodSpec? Following questions for that API

• How would we easily get more than one schema (to save making multiple requests)?
• How would we get updates to those schemas? Not quite sure we would need this, is there anything permissions based that would necessitate the UI requiring updates (for the specific spec, for the root schema objects we'd get updates as usual)?

[MbolotSuse]

Would it be possible to add some examples, given the features in the current PR, of fetching schemas that have been dropped? So for instance how would we get the PodSpec?

In the current implementation, you would not be able to get them through steve. You could still get them through kuberenetes (https://kubernetes.io/docs/concepts/overview/kubernetes-api/ - in short /openapi/v3 or /openapi/v2 on the k8s server), but as noted above this is slow.

This change (which was never merged) was made under the theory that the UI didn't need field definitions and mostly needed schemas for permissions checks, which aren't needed for subresources (like Pod specs) outside of a narrow few cases (and I don't know if steve works with these today). This appeared to not be true since the UI uses these resources for helm charts and the like, but I'm guessing that the investigation ticket was opened since we didn't know to what extent it was needed.

As far as your other questions go, that will depend on the implementation of this k8s side. We won't offer any features outside of what k8s already applies by default.

[richard-cox]

Is it possible to fetch schemas via openapi though? Think I'm conceptually missing something

I've looked through openapi/v3/api/v1 and there's nothing in paths, just the references in component/schemas.

Closest reference i could get to was /openapi/v3/api/v1#components/schemas/io.k8s.api.core.v1.PodSpec (which given the fragment implies it should be in a website).

Other things I've tried

• https://rancher.richardcox.dev/openapi/v3/api/v1/components/schemas/io.k8s.api.core.v1.PodSpec
• https://rancher.richardcox.dev/openapi/v3/api/v1/schemas/io.k8s.api.core.v1.PodSpec
• https://rancher.richardcox.dev/openapi/v3/api/v1/schemas/PodSpec

[MbolotSuse]

Those paths are for openapi v3. As noted in the k8s docs, that is currently a feature in beta state that is disabled by default. So even if that did work, I don't think you'll be able to consistently use those endpoints.

However, you can use those endpoints to get the information that you are looking for. openapi/v3/api/v1 has the pod schema inside of it - I checked, it's in the large map returned at that endpoint. I'm unsure if you can get only the schema for pods, not super familiar with this functionality.

You'll need to use the openapi v2 endpoints - /openapi/v2/ gives you all of the schemaas (including the pod spec). I didn't, on quick examination, see a way to request only 1 schema, and the docs seem to indicate that this is the aggregated schema. If I uses kubectl explain pod.spec -v 8, kubectl makes a request to this endpoint and seems to load in all of these values.

[richard-cox]

Sorry, understood.

• via openapi UI can fetch ALL schemas
• possible alternative have a new endpoint to fetch them (tbd whether this is the original schemas endpoint or new resources)

Will discuss internally and come up with a propasal

[MbolotSuse]

Thanks @richard-cox . I would also appreciate it if you could explain the various areas/things that you use the schemas for - part of the reason that the original implementation didn't work is that we assumed that the UI wasn't using the field definitions, which wasn't true. Knowing what fields are used and why can help us optimize on the backend.

[richard-cox]

I need to go back and understand each use case from https://github.com/rancher/dashboard/issues/7716#issuecomment-1359500983, but i think the endpoints below would be a good starting point

• Endpoint to fetch schemas without their resourceFields (some would be secondary schemas)
  • for simplicity this would be the current schemas endpoint / resource type, which i think has already been done via the linked PR
  • These schemas will be watched via socket and are permission dependent (visibility of schema, schema properties, etc)
• New endpoint to fetch a specific schema including it's resourceFields
  • when resourceField is a secondary schema it includes the full definition of itself and any child schemas
  • schema requested could be a primary or secondary
  • schemas in this context are not required to be updated and have no permission content

Given the usages in https://github.com/rancher/dashboard/issues/7716#issuecomment-1359500983

• Require specific secondary schema - http request to new endpoint made on demand. cached only in context of page
• Required to guide user write yaml - http request to new endpoint made on demand. cached only in context of page
• schema's remain unchanged

[gaktive]

Related backend now pushed to Q4 at least.

[richard-cox]

We need to investigate how resource validation against schema's works and if it's impacted by this change

[richard-cox]

I've played around with things given the image Michael very kindly supplied.

I had a look how we would handle only the change to v1/schemas as mentioned in https://github.com/rancher/rancher/issues/41301. This involves fetching schemas via Openapi.

• Hitting openapi/v2 fetches all schemas.
  • Size of vanilla, just installed rancher's v1/schema is 370kB compress, 8.7MB on disc. This is quite meaty
  • With openapi/v2 we're unable to get a specific schema (it's in v3 but that's not prod ready). https://github.com/rancher/dashboard/issues/7716#issuecomment-1485576248
• Schemas returned by openapi/v2 are in a very different format to v1/schemas
  • Pretty much only contain the resourceField info
  • schema name is very different (Neil has already created a nice way to get our version of the name from their schema)
  • inline references to other schemas are different
    • metadata $ref: "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta"
    • metadata": { "type": "io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta", "create": true, "update": true },

So two options

1. MVP
   • We use new v1/schemas
     • This does not include resourceFields
     • This does not include schemas that would have only been referenced by secondary fields
   • We don't use proposed v1/schemaDefinitions, but openapi/v2 instead
     • This provides the missing resourceFields
     • This is LARGE so should take many steps to reduce impact
       • we should avoid the vuex store with these, or Object.freeze them, or both?
       • we should investigate crossplane
     • We will convert the openapi2 format into rancher schema to avoid mass changes in code
2. Preferred Solution
   • We use new v1/schemas same as MVP
   • We use new v1/schemaDefinitions
     • referenced in https://github.com/rancher/rancher/issues/41301
     • UI requests specific schema
     • Response contains specific schema AND it's secondary schemas
     • for instance request for v1/schemaDefinitions/pods returns schema for pod, objectmeta, pod spec, pod status, etc
       • this can be inline / nested in pod object
       • if they're not inline, the request to /<specific resource> needs to be an array of schemas
   • This removes the need to fetch openapi schemas which should greatly improve performance
   • This can also then be used in the new kube explain feature in 2.8.0 (which is used in a specific, deliberate way rather than code that will be called often as per this issue)

[richard-cox]

Currently waiting on a new build from Michael (discussed in slack dm)

[richard-cox]

Starting info for release notes.

Change introduces updates to core components that cater for breaking api changes in 2.9.0.

This means extensions using older versions of shell use previous components without this update which can result in some features not working (for example creating a value from yaml will not display content or saving resources will not validate against schemas on save).

Uses will need to rebuilt their extension with a version of shell at or higher than TBD.

[richard-cox]

#10961, #11173 should be tested at the same time

[sunilarjun]

Starting info for release notes.

Change introduces updates to core components that cater for breaking api changes in 2.9.0.

This means extensions using older versions of shell use previous components without this update which can result in some features not working (for example creating a value from yaml will not display content or saving resources will not validate against schemas on save).

Uses will need to rebuilt their extension with a version of shell at or higher than TBD.

Hello @richard-cox! I am handling release notes for v2.9.0 and was wondering if you could expand on the last section where users will "need to rebuild the extension with a version of the shell at or higher than TBD", was the TBD determined? Any help appreciated, thank you!

[yonasberhe23]

Tested in

• v2.9-556e3f04c336ac4dbe8d0170fecdcf515973e65b-head
• master e108b95

all scenarios in attached PR covered.

[richard-cox]

@sunilarjun For this feature.. Improve UI performance by exercising new API feature to independently fetch schemes related to the primary resource (please reword if appropriate!)

In addition we need somewhere in the release notes to explain a requirement for UI Extension developers to update their extensions (update @rancher/shell dependency and chart annotations, and release a new version) given changes within 2.9.0. @aalves08 Do we have a blurp for the release notes for above?

[aalves08]

Here's the page that I created for our documentation: https://extensions.rancher.io/extensions/rancher-2.9-support

[richard-cox]

Thanks Alex

@sunilarjun I've added some review comments to https://github.com/rancherlabs/release-notes/pull/390#pullrequestreview-2197091133. So it's those plus feature text from https://github.com/rancher/dashboard/issues/7716#issuecomment-2247561592

[sunilarjun]

Thanks for the review and responses @richard-cox and @aalves08! Addressing the feedback currently and will respond in the release notes thread with updates.