json-schema-ref-parser
json-schema-ref-parser copied to clipboard
APIDevTools
bundle() creates a invalid $ref
We are using the json-schema-ref-parser library to flatten / dereference our schemas. As we need to return them over an API, we use bundle() as sometimes circular references can occur in the given JSON schemas.
One user just discovered that his schema got invalid after using bundle() as four refs got replaced with a non-valid ref, pointing to a non-existing path. These schemas are quite complicated with many allOfs. I reproduced the broken schema and removed everything so just the invalid ref and needed surroundings are given:
{
"type": "object",
"allOf": [
{
"description": "REMOVED for better readbility"
},
{
"type": "object",
"properties": {
"payload": {
"type": "array",
"items": {
"allOf": [
{
"description": "REMOVED for better readbility"
},
{
"type": "object",
"properties": {
"reservationActionMetaData": {
"allOf": [
{
"allOf": [
{
"type": "object",
"properties": {
"supplierPriceElements": {
"allOf": [
{
"description": "REMOVED for better readbility"
},
{
"type": "object",
"properties": {
"purchaseRate": {
"allOf": [
{
"type": "object",
"required": [
"amount"
],
"properties": {
"amount": {
"type": "number"
}
}
},
{
"type": "object",
"properties": {
"inDetail": {
"type": "object",
"properties": {
"perDate": {
"type": "array",
"items": {
"type": "object",
"properties": {
"amount": {
"$ref": "#/allOf/1/properties/payload/items/allOf/1/properties/reservationActionMetaData/allOf/0/allOf/0/properties/supplierPriceElements/allOf/1/properties/fee/properties/modificationFee/properties/amount"
}
}
}
}
}
}
}
}
]
},
"fee": {
"type": "object",
"properties": {
"modificationFee": {
"$ref": "#/allOf/1/properties/payload/items/allOf/1/properties/reservationActionMetaData/allOf/0/allOf/0/properties/supplierPriceElements/allOf/1/properties/purchaseRate/allOf/0"
}
}
}
}
}
]
}
}
},
{
"description": "REMOVED for better readbility"
}
]
},
{
"description": "REMOVED for better readbility"
}
]
}
}
}
]
}
}
}
}
]
}
One ref (amount) points to a path (modificationFee) which contains another ref - and this contains the needed data. But the compiler cannot compile this (e.g. https://www.jsonschemavalidator.net/).
Error message:
Could not resolve schema reference '#/allOf/1/properties/payload/items/allOf/1/properties/reservationActionMetaData/allOf/0/allOf/0/properties/supplierPriceElements/allOf/1/properties/fee/properties/modificationFee/properties/amount'. Path 'allOf[1].properties.payload.items.allOf[1].properties.reservationActionMetaData.allOf[0].allOf[0].properties.supplierPriceElements.allOf[1].properties.purchaseRate.allOf[1].properties.inDetail.properties.perDate.items.properties.amount', line 59, position 123.
I can try to provide some stripped down schemas, but this could take some time. We have three schemas which reference themselves like A -(one ref)-> B -(multiple refs)-> C.
Maybe you have already an idea how this could happen?
If you could provide the schemas that cause the issue we could debug it and add a test case
Sure, I'll try to deliver the schemas this week. Need to anonymize them first. Thanks in advance.
I finally found the time to strip down the schemas. If you bundle() schemaA, the bundled schema will contain the invalid $ref as described in my first message.
(The schemas are quite complex, stripped down over 1000 lines. If anything does not work, let me know.)
Here's the bundled schema:
{
"$id": "http://example.com/schemaA/1.0",
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"allOf": [
{
"type": "object",
"required": [
"eventId",
"payload"
],
"properties": {
"eventId": {
"type": "string"
}
}
},
{
"type": "object",
"properties": {
"payload": {
"type": "array",
"items": {
"allOf": [
{
"type": "object"
},
{
"type": "object",
"properties": {
"reservationActionMetaData": {
"allOf": [
{
"allOf": [
{
"type": "object",
"required": [
"supplierPriceElements"
],
"properties": {
"supplierPriceElements": {
"allOf": [
{
"required": [
"type"
],
"properties": {
"type": {
"type": "string"
}
}
},
{
"type": "object",
"required": [
"purchaseRate"
],
"properties": {
"purchaseRate": {
"allOf": [
{
"type": "object",
"required": [
"amount",
"currency"
],
"properties": {
"amount": {
"type": "number",
"format": "float"
},
"currency": {
"type": "string",
"minLength": 1
}
}
},
{
"type": "object",
"properties": {
"inDetail": {
"type": "object",
"properties": {
"perDate": {
"type": "array",
"items": {
"type": "object",
"properties": {
"date": {
"type": "string",
"format": "date"
},
"amount": {
"$ref": "#/allOf/1/properties/payload/items/allOf/1/properties/reservationActionMetaData/allOf/0/allOf/0/properties/supplierPriceElements/allOf/1/properties/fee/properties/modificationFee/properties/amount"
},
"detailedPriceInformation": {
"type": "array",
"items": {
"type": "object",
"properties": {
"amount": {
"$ref": "#/allOf/1/properties/payload/items/allOf/1/properties/reservationActionMetaData/allOf/0/allOf/0/properties/supplierPriceElements/allOf/1/properties/fee/properties/modificationFee/properties/amount"
},
"paxId": {
"type": "string"
},
"inDetail": {
"type": "object",
"properties": {
"rate": {
"$ref": "#/allOf/1/properties/payload/items/allOf/1/properties/reservationActionMetaData/allOf/0/allOf/0/properties/supplierPriceElements/allOf/1/properties/fee/properties/modificationFee/properties/amount"
},
"board": {
"$ref": "#/allOf/1/properties/payload/items/allOf/1/properties/reservationActionMetaData/allOf/0/allOf/0/properties/supplierPriceElements/allOf/1/properties/fee/properties/modificationFee/properties/amount"
},
"taxes": {
"type": "array",
"items": {
"$ref": "#/allOf/1/properties/payload/items/allOf/1/properties/reservationActionMetaData/allOf/0/allOf/0/properties/supplierPriceElements/allOf/1/properties/purchaseRate/allOf/1/properties/inDetail/properties/perDate/items/properties/detailedPriceInformation/items/properties/inDetail/properties/fees/items"
}
},
"fees": {
"type": "array",
"items": {
"type": "object",
"properties": {
"id": {
"type": "string"
},
"amount": {
"$ref": "#/allOf/1/properties/payload/items/allOf/1/properties/reservationActionMetaData/allOf/0/allOf/0/properties/supplierPriceElements/allOf/1/properties/fee/properties/modificationFee/properties/amount"
}
}
}
},
"supplements": {
"type": "array",
"items": {
"$ref": "#/allOf/1/properties/payload/items/allOf/1/properties/reservationActionMetaData/allOf/0/allOf/0/properties/supplierPriceElements/allOf/1/properties/purchaseRate/allOf/1/properties/inDetail/properties/perDate/items/properties/detailedPriceInformation/items/properties/inDetail/properties/fees/items"
}
},
"salesOfferIds": {
"type": "array",
"items": {
"$ref": "#/allOf/1/properties/payload/items/allOf/1/properties/reservationActionMetaData/allOf/0/allOf/0/properties/supplierPriceElements/allOf/1/properties/purchaseRate/allOf/1/properties/inDetail/properties/perDate/items/properties/detailedPriceInformation/items/properties/inDetail/properties/fees/items"
}
}
}
}
}
}
}
}
}
},
"perPax": {
"type": "array",
"items": {
"type": "object",
"properties": {
"id": {
"type": "string"
},
"amount": {
"$ref": "#/allOf/1/properties/payload/items/allOf/1/properties/reservationActionMetaData/allOf/0/allOf/0/properties/supplierPriceElements/allOf/1/properties/fee/properties/modificationFee/properties/amount"
},
"salesOfferIds": {
"type": "array",
"items": {
"type": "string"
}
}
}
}
}
}
}
}
}
]
},
"fee": {
"type": "object",
"properties": {
"modificationFee": {
"$ref": "#/allOf/1/properties/payload/items/allOf/1/properties/reservationActionMetaData/allOf/0/allOf/0/properties/supplierPriceElements/allOf/1/properties/purchaseRate/allOf/0"
},
"cancellationFee": {
"$ref": "#/allOf/1/properties/payload/items/allOf/1/properties/reservationActionMetaData/allOf/0/allOf/0/properties/supplierPriceElements/allOf/1/properties/purchaseRate/allOf/0"
}
}
}
}
}
]
},
"type": {
"type": "string"
}
}
},
{
"type": "object",
"required": [
"test"
],
"properties": {
"test": {
"type": "string"
}
}
}
],
"properties": {
"type": {
"type": "string"
}
}
},
{
"type": "object",
"required": [
"test"
],
"properties": {
"test": {
"type": "string"
}
}
}
]
}
}
}
]
}
}
}
}
]
}
Thanks in advance! :-)
This looks like a bug with https://www.jsonschemavalidator.net/ - what part of the json schema is invalid? It's a $ref that points to another $ref, with a few layers of indirection, but it eventually resolves to the right value.
A referenced path is valid post all dereferencing - in your example, #/allOf/1/properties/payload/items/allOf/1/properties/reservationActionMetaData/allOf/0/allOf/0/properties/supplierPriceElements/allOf/1/properties/fee/properties/modificationFee/properties/amount is actually a perfectly valid path, once you dereference all the values.
Here's the smallest reproducible example that bugs out on that site: SchemaA:
{
"$id": "schemaA/1.0",
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"$ref": "schemaB.json#/definitions/SupplierPriceElement"
}
SchemaB
{
"$id": "schemaC/1.0",
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"SupplierPriceElement": {
"type": "object",
"properties": {
"purchaseRate": {
"$ref": "#/definitions/InDetailParent"
},
"fee": {
"$ref": "#/definitions/AllFees"
}
}
},
"AllFees": {
"type": "object",
"properties": {
"modificationFee": {
"$ref": "#/definitions/MonetaryAmount"
}
}
},
"MonetaryAmount": {
"type": "object",
"properties": {
"amount": {
"$ref": "#/definitions/Amount"
}
}
},
"Amount": {
"type": "number",
"format": "float"
},
"InDetailParent": {
"allOf": [
{
"$ref": "#/definitions/MonetaryAmount"
},
{
"type": "object",
"$ref": "#/definitions/Amount"
}
]
}
}
}
This produces
{
"$id": "schemaA/1.0",
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"purchaseRate": {
"allOf": [
{
"type": "object",
"properties": {
"amount": {
"type": "number",
"format": "float"
}
}
},
{
"type": "object",
"$ref": "#/properties/fee/properties/modificationFee/properties/amount"
}
]
},
"fee": {
"type": "object",
"properties": {
"modificationFee": {
"$ref": "#/properties/purchaseRate/allOf/0"
}
}
}
}
}
The site considers this invalid, but it considers the exact same spec with the keys rearranged in order as valid
{
"$id": "schemaA/1.0",
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"fee": {
"type": "object",
"properties": {
"modificationFee": {
"$ref": "#/properties/purchaseRate/allOf/0"
}
}
},
"purchaseRate": {
"allOf": [
{
"type": "object",
"properties": {
"amount": {
"type": "number",
"format": "float"
}
}
},
{
"type": "object",
"$ref": "#/properties/fee/properties/modificationFee/properties/amount"
}
]
}
}
}
Even though the files are identical:
Please let me know if I got something wrong here
Hello @jonluca,
Thank you for your response. I recently tested the schema you mentioned (which was not functioning) and also the “re-arranged” schema (which worked) using www.jsonschemavalidator.com. You are correct; when rearranged, it seems to work like magic. Unfortunately, this behavior is only observed in the context of this specific JSON validator.
I also tested it
- in JavaScript / NodeJS with AJV (https://ajv.js.org/),
- in Golang with https://github.com/santhosh-tekuri/jsonschema, and
- as little web application provided by https://github.com/ISAITB/json-validator
with the non-re-arranged and the re-arranged schemas.
All compilers / validators have encountered the following errors with both schemas:
| Compiler / Validator | Error Message |
|---|---|
| AJV | "can't resolve reference #/properties/fee/properties/modificationFee/properties/amount from id schemaA/1.0" |
| https://github.com/santhosh-tekuri/jsonschema | jsonschema https://example.com/schemaA/1.0 compilation failed: https://example.com/schemaA/1.0#/properties/fee/properties/modificationFee/properties/amount not found |
| https://github.com/ISAITB/json-validator | An error occurred during the validation: /properties/fee/properties/modificationFee: Reference /properties/fee/properties/modificationFee/properties/amount cannot be resolved. |
I also attempted to locate relevant information within the Draft 7 specifications, but unfortunately, I could not find any specific details on this topic. It appears that perhaps this usage was never originally intended, as all the compilers and validators I tested have encountered failures. Alternatively, it’s possible that various projects have misinterpreted the specifications or overlooked implementing this particular feature.
As numerous compilers and validators struggle to interpret this bundled schema, is there a way to override this behavior? I’ll revisit the specifications to see if I can find any relevant information on this topic.
Thank you in advance.
Hello @jonluca,
is it possible to re-open this ticket and discuss the findings I provided with the different compilers? I still think the created $refs are invalid, as they are not directly point to the supposed value.
Thanks in advance.
Hello @jonluca,
thank you for re-opening the issue. While searching through the specifications this morning, I discovered some links to the official JSON Schema Spec Github project and thought: Maybe I could just ask them, as they are the officials and maybe have an answer to this topic, as this would make it easier for us both.
Here's my question (issue) I opened at their project. And someone also just answered: https://github.com/json-schema-org/json-schema-spec/issues/1508
As it seems, the $ref is not correct like this. Would it be possible to fix this somehow?
Thanks in advance.
hello
i came across this issue because i encounter the same undesired behavior, like others (see https://github.com/hey-api/openapi-ts/issues/1135).
There is an option to disable the faulty ref internal bundler ?
I encountered the same issue.
In my case schema C references property schemas A and B
properties:
p1:
$ref: /A.schema
p2:
$ref: /B.schema
Schema C
while B references A
allOf:
- $ref: /A.schema
Schema B
Now B and A were dereferenced at #/properites/p2 but the reference /A.schema within the dereferenced schema B was changed to #/properties/p1
It seems to be related with the reference sorting. Disable the criteria extended here
https://github.com/APIDevTools/json-schema-ref-parser/blob/588197e114d9a8ef67dd78181cfa05e48e4de742/lib/bundle.ts#L207-L209
solved the issue at least in this particular case.
Now A was dereferenced at #/properites/p1 and B in #/properites/p2 and the reference /A.schema within the dereferenced schema B correctly pointing to #/properties/p1
Not sure if this was just luck. A more solid workaround would be skipping the reference remapping completely and just dump the dereferenced schemas in place (except circular refs)
