kin-openapi icon indicating copy to clipboard operation
kin-openapi copied to clipboard

cache loaded remote schemas

Open bonnyr opened this issue 5 years ago • 2 comments

when resolving components for loaded schemas and when external resolution is enabled, kin-openapi will read the remote schema, parse it and resolve it in order to satisfy the reference attribute in the calling schema, but it does so every time a reference needs resolving leading to unnecessary network traffic request for the same remote schema.

For example:

components:
  schemas:
    AnotherTestSchema:
      type: object
      properties:
        ref1:
          "$ref": http://localhost:7965/components.openapi.yml#/components/schemas/CustomTestSchema
        ref2:
          "$ref": http://localhost:7965/components.openapi.yml#/components/schemas/Name

In the above case, external resolution for components.openapi.yml is performed when parsing the fragment above only to be done again for ref2. This is recursive and if components.openapi.yml includes other references they are resolved the same way.

This very quickly leads to major network traffic which at the very least is unnecessary.

This pull request caches the loaded schemas in the swagger loader and returns resolved schemas from the cache once they are loaded.

Added tests to ensure the schemas are loaded only once.

bonnyr avatar Sep 24 '19 18:09 bonnyr

I understand this can be a major PITA and am very curious how this is supposed to be handled by OpenAPI / JSON Schema specs? Surely caching of HTTP content should at least honor the HTTP cache headers? Are remote schemas expected to be assumed to never change?

Introducing remote refs immutability may break things, introduce races... so I'm leaning towards "these just have to be immutable". I just can't find the part of the documentation mentioning this?

fenollp avatar Oct 02 '19 10:10 fenollp

@bonnyr Hi! I think this should get in. Do you mind rebasing? Thanks

Actually there now is a swaggerLoader.LoadSwaggerFromURIFunc so just adding an implementation of LoadSwaggerFromURIFunc that relies on an outside cache var (maybe behind a mutex) sounds like an appropriate solution and would make for a great example that others can derive from for their own cache semantics needs.

fenollp avatar Feb 18 '21 10:02 fenollp

See https://pkg.go.dev/github.com/getkin/kin-openapi/openapi3#ReadFromURIFunc and specifically https://pkg.go.dev/github.com/getkin/kin-openapi/openapi3#URIMapCache which is used by default:

var DefaultReadFromURI = URIMapCache(ReadFromURIs(ReadFromHTTP(http.DefaultClient), ReadFromFile))

fenollp avatar Sep 14 '22 14:09 fenollp