Support JSON $refs

[stefanthorpe]

The adoption of refs is growing. Would be great if JQ could collect refs and allow querying against them.

Details on refs

https://github.com/whitlockjc/json-refs

[emanuele6]

What are we supposed to implement?

I guess we could add a resolverefs/0 function that resolves JSON References, but...

Afaict, a JSON Reference is an object with only a $ref field that has a string value that represents a URI to a JSON file with a JSON pointer-encoded path to a value in that JSON file as fragment.

For example:

{ "$ref": "http://example.com/example.json#/foo/bar" }

What are we supposed to do with that? Use a library like libcurl to download http://example.com/example.json, parse it as a JSON file, and then extract the value pointed by that JSON reference otherwise throw an error if the JSON file or the JSON pointer is invalid?

Only support JSON references with relative URIs like .#/a/b to get .a.b of the current JSON file? What if the input file is a pipe so that cannot be done? Should it try to do that and fail? Should jq remember what it read and resolve .a.b from what it remembers? Should it resolve relative URIs from the most recently read input? from the input that is currently being processed?

Maybe it should support file:///path/to/foo.json#/a/b URIs to get a value from another local file? That would be cool, but adding support for JSON References only for this seems weird.

That RFC is still a draft after almost 6 years, would this feature be useful to anyone?

[nicowilliams]

Eventually we might add support for things like fetching resources over HTTP. For now, no.

[nicowilliams]

What are we supposed to do with that? Use a library like libcurl to download http://example.com/example.json, parse it as a JSON file, and then extract the value pointed by that JSON reference otherwise throw an error if the JSON file or the JSON pointer is invalid?

That sounds right. IF we wanted to implement this.

[nicowilliams]

Only support JSON references with relative URIs like .#/a/b to get .a.b of the current JSON file? What if the input file is a pipe so that cannot be done? Should it try to do that and fail? Should jq remember what it read and resolve .a.b from what it remembers? Should it resolve relative URIs from the most recently read input? from the input that is currently being processed?

We could do it in a function to resolve JSONPointer references, but not while jq is parsing a JSON, no way.

[pkoppstein]

Being able to fetch a URL programmatically would be very handy. To quote @nicowilliams with regard to some work in that direction:

Hey, this is very cool! Thanks! I'll do my best to integrate this soon.

Alas, that was in 2015. https://github.com/jqlang/jq/issues/650#issuecomment-67922555

Perhaps a new Issue can be opened that would encourage the incorporation of this particular functionality without it being lost in the weeds of $ref.

[FelixZY]

Related issue from yq: https://github.com/mikefarah/yq/issues/1004

And suggested solution: https://mikefarah.gitbook.io/yq/operators/load#replace-all-nodes-with-referenced-file

---

EDIT

I was able to construct a yq query to replace $ref with local refs in a json file. The solution can probably be amended to work with external refs as well via load.

https://github.com/mikefarah/yq/issues/1004#issuecomment-2093552089

[wader]

As fq has a open function i had to try :)

$ cat a.json
{
  "a": {
    "$ref": "b.json"
  }
}
$ cat b.json
{
  "test": 123
}
$ fq 'def load($file): $file | open | fromjson; (.. | select(has("$ref")?)) |= load(.["$ref"])' a.json
{
  "a": {
    "test": 123
  }
}

... and i should probably document it better and its security concerns 😬