ResolverError while processing bundled schema

[PopGoesTheWza]

We have to process a bunch of json schemas produced by another team and bundled using @hyperjump/json-schema.

For reference (pun intended) here is a simple sample of such bundles.

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "http://common-schemas.redacted.com",
  "type": "object",
  "properties": {
    "dateInMilliseconds": { "$ref": "#/definitions/DateInMilliseconds" },
    "monetary": { "$ref": "#/definitions/Monetary" },
    "reference": { "$ref": "#/definitions/Reference" },
    "referenceWithExternalId": {
      "$ref": "#/definitions/ReferenceWithExternalId"
    }
  },
  "required": [
    "dateInMilliseconds",
    "monetary",
    "reference",
    "referenceWithExternalId"
  ],
  "additionalProperties": false,
  "title": "CommonTypes",
  "description": "Bunndle of common types",
  "definitions": {
    "DateInMilliseconds": {
      "$ref": "http://common-schemas.redacted.com/date-in-milliseconds"
    },
    "Monetary": {
      "$ref": "http://common-schemas.redacted.com/monetary"
    },
    "Reference": {
      "$ref": "http://common-schemas.redacted.com/reference"
    },
    "ReferenceWithExternalId": {
      "$ref": "http://common-schemas.redacted.com/reference-with-external-id"
    },
    "http://common-schemas.redacted.com/date-in-milliseconds": {
      "$id": "http://common-schemas.redacted.com/date-in-milliseconds",
      "title": "DateInMilliseconds",
      "description": "A UTC datetime in milliseconds",
      "type": "integer"
    },
    "http://common-schemas.redacted.com/monetary": {
      "$id": "http://common-schemas.redacted.com/monetary",
      "title": "Monetary",
      "description": "An amount of money in a specific currency.",
      "type": "object",
      "properties": {
        "amount": {
          "type": "number",
          "description": "The net monetary value. A negative amount denotes a debit; a positive amount a credit."
        },
        "currency": {
          "type": "string",
          "description": "The [ISO 4217 currency code](https://en.wikipedia.org/wiki/ISO_4217) for this monetary value. This is always upper case ASCII.",
          "minLength": 3,
          "maxLength": 3
        }
      },
      "required": ["amount", "currency"],
      "additionalProperties": false
    },
    "http://common-schemas.redacted.com/reference": {
      "$id": "http://common-schemas.redacted.com/reference",
      "title": "Reference",
      "description": "Reference to an API object",
      "type": "object",
      "additionalProperties": false,
      "properties": {
        "id": {
          "type": "string",
          "description": "Well known global unique identifier",
          "minLength": 1
        },
        "lastModif": { "$ref": "#/definitions/DateInMilliseconds" }
      },
      "required": ["id", "lastModif"],
      "definitions": {
        "DateInMilliseconds": {
          "$ref": "http://common-schemas.redacted.com/date-in-milliseconds"
        }
      }
    },
    "http://common-schemas.redacted.com/reference-with-external-id": {
      "$id": "http://common-schemas.redacted.com/reference-with-external-id",
      "title": "ReferenceWithExternalId",
      "description": "Reference to an API object which has an `externalId' property",
      "type": "object",
      "additionalProperties": false,
      "properties": {
        "externalId": {
          "type": ["string", "null"],
          "description": "Well known global unique identifier from an external data source",
          "minLength": 1
        },
        "id": {
          "type": "string",
          "description": "Well known global unique identifier",
          "minLength": 1
        },
        "lastModif": { "$ref": "#/definitions/DateInMilliseconds" }
      },
      "required": ["externalId", "id", "lastModif"],
      "definitions": {
        "DateInMilliseconds": {
          "$ref": "http://common-schemas.redacted.com/date-in-milliseconds"
        }
      }
    }
  }
}

Above bundle references four schemas which are plainly defined under #/definitions with matching $ids.

• "$id": "http://common-schemas.redacted.com/date-in-milliseconds"
• "$id": "http://common-schemas.redacted.com/monetary"
• "$id": "http://common-schemas.redacted.com/reference"
• "$id": "http://common-schemas.redacted.com/reference-with-external-id"

Whenever we attempt to process it (initially via json-schema-to-typescript or using json-schema-ref-parser's dereference() function) we get the bellow ResolverError.

/Users/redacted/dev/redacted/node_modules/.pnpm/@[email protected]/node_modules/@apidevtools/json-schema-ref-parser/dist/lib/resolvers/http.js:123
        throw new errors_js_1.ResolverError((0, ono_1.ono)(err, `Error downloading ${u.href}`), u.href);
              ^

ResolverError: Error downloading http://common-schemas.redacted.com/date-in-milliseconds 
fetch failed
    at download (/Users/redacted/dev/redacted/node_modules/.pnpm/@[email protected]/node_modules/@apidevtools/json-schema-ref-parser/dist/lib/resolvers/http.js:123:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5) {
  code: 'ERESOLVER',
  source: 'http://common-schemas.redacted.com/date-in-milliseconds',
  path: null,
  toJSON: [Function: toJSON],
  [Symbol(nodejs.util.inspect.custom)]: [Function: inspect]
}

We are probably missing something obvious, but couldn't find how to prevent "bundled" references to be (wrongfully) processed by the http parser while already referenced under #/definitions.

Any advice?

Thanks in advance,

[jonluca]

Does creating a custom http resolver fix this?

https://github.com/APIDevTools/json-schema-ref-parser/blob/main/test/specs/resolvers/resolvers.spec.ts

Like if you provide your own version of canRead and readFile for http?

[PopGoesTheWza]

Thanks for the hint @jonluca

Even though I had give it a good read, I missed that canRead and read properties both accept a function with the (file: FileInfo, callback: Callback, $refs: $Refs) => signature. The current TypeScript definition files do not cover this clearly.

So if I read your suggestion correctly, we should implement a custom resolver that is called before the standard http.

• resolve.custom.canRead to check if the file parameter is a reference already defined within the bundle (using the $refs parameter I assume)?
• resolve.custom.read to return the corresponding definition (there again using the $refs parameter)

Is this what you have in mind?

[PopGoesTheWza]

@jonluca I have investigated my issue and completed a dirty work-around using the below resolvers with @apidevtools/json-schema-ref-parser 11.9.3.

    resolve: {
      definitions: {
        order: 1,
        canRead(file: FileInfo, callback: Callback, $refs: $Refs) {
          console.log('resolve.definitions.canRead', typeof file, typeof callback, typeof $refs);

          return true;
        },
        read(file: FileInfo, callback: Callback, $refs: $Refs) {
          console.log('resolve.definitions.read', typeof file, typeof callback, typeof $refs);

          const {url: $id} = file;
          const {definitions} = $refs._root$Ref.value;

          if (definitions) {
            const definition = definitions[$id];
            if (definition) {
              return JSON.stringify(definition);
            }
          }

          return 'bad resolve';
        },
      },
      http: {
        order: 2000,
        canRead(file: FileInfo, callback: Callback, $refs: $Refs) {
          console.log('resolve.http.canRead', typeof file, typeof callback, typeof $refs);

          return false;
        },
      },
    },

When processing the schema from above original post, the console outputs the following.

resolve.http.canRead object undefined undefined
resolve.definitions.canRead object undefined undefined
resolve.definitions.read object function object
resolve.http.canRead object undefined undefined
resolve.definitions.canRead object undefined undefined
resolve.definitions.read object function object
resolve.http.canRead object undefined undefined
resolve.definitions.canRead object undefined undefined
resolve.definitions.read object function object
resolve.http.canRead object undefined undefined
resolve.definitions.canRead object undefined undefined
resolve.definitions.read object function object

Some remarks:

• resolvers' order property as no incidence. Looking at the source code, it looks like only plugins are sorted on order and not resolvers. This has the http resolver being triggered before my custom definitions resolver.
• the $Refs type definition is not available so I use lazy type $Refs = { _root$Ref: {value: JSONSchema}; };
• Only the read function gets the $Ref as third argument. Since canRead does not, it makes it difficult to check my use case (i.e. if the refrence exists within definitions)

[jonluca]

I'll push an update that passes the $refs to the canRead function as well.

It looks like the sort order for canRead is working properly though, they are sorted from smallest to largest, and it shortCircuits once one of them can read it. The string 'bad resolve' is evaluating to be truthy, so then it calls read. This might've been fixed at some point between this issue being created and me replying though.

Were you able to figure out the fetch issue? It looks to me like it's most likely a networking error or bot detection or something of the like.