Implement dereference functionality

[carlesarnal]

Implement the dereference parameter support in the API. This support will be gradual since the dereference support is specific for each artifact type.

• [ ] https://github.com/Apicurio/apicurio-registry/issues/4256
• [ ] https://github.com/Apicurio/apicurio-registry/issues/4257

[ai-zelenin]

may i ask when this feature will be implemented? we plan use debezium with apicurio-registry but without dereference we unable use avro decoder in golang/python(need both)

[carlesarnal]

Unfortunately, there is no timeframe for it. It's for sure on our backlog and we're working on a wide range of improvements for our artifact references support, but we have no dates yet,

[EricWittmann]

I would like to start working on this soon. @MikeEdgar this is the motivation for migrating/porting the dereference support in apicurio-data-models. But we will need to do the same for the other types we support (Avro, Protobuf, JSON Schema).

[seb-pereira]

Hi, is thedereference query parameter documented in the REST API expected to work with latest version 2.5.x with Avro schemas?

I am following the example from the user documentation and I can not find an endpoint that returns a schema with resolved references when the query parameter dereference is set to true. I tried:

• by global Id (ex: /apis/registry/v2/ids/globalIds/14?dereference=true)
• by artifact Id (ex: /apis/registry/v2/groups/group/artifacts/Item?dereference=true)

[davidradl]

@seb-pereira I found the same , I upgraded to 2.5.9 which says it has support for dereference for Avro but this rest end point does not resolve at that level either.

@EricWittmann @carlesarnal please could you review

[carlesarnal]

I just re-tested this and I get the expected result.

I have a few artifacts registered, each with the expected references, for instance, I'm using the example here. If I let the maven-plugin do the registration for me, this results in 4 artifacts created:

1. Exchange, an enum, no references to other artifacts, globalId 1.
2. TradeKey, a record, with a reference to Exchange, globalId 2.
3. TradeValue, a record, with a reference to Exchange, globalId 3.
4. TradeRaw, a record, with two references, to TradeKey and to TradeValue, globalId 4.

If I get the content of TradeRaw straight, I get the expected result, the original content of the artifact in the filesystem:

With this path (artifactId, the result by globalId is the same) http://localhost:8080/apis/registry/v2/groups/avro-maven-with-references-auto/artifacts/TradeRaw:

{
  "namespace": "com.kubetrade.schema.trade",
  "type": "record",
  "name": "TradeRaw",
  "fields": [
    {
      "name": "tradeKey",
      "type": "com.kubetrade.schema.trade.TradeKey"
    },
    {
      "name": "value",
      "type": "com.kubetrade.schema.trade.TradeValue"
    },
    {
      "name": "symbol",
      "type": "string"
    },
    {
      "name": "payload",
      "type": "string"
    }
  ]
}

Now, if I try to get the content dereferenced using the same endpoint but with dereference set to true, the result is the expected one, the content dereferenced (nested records inlined): Using the path (the result using the globalId is the same) http://localhost:8080/apis/registry/v2/groups/avro-maven-with-references-auto/artifacts/TradeRaw?dereference=true:

{
  "type": "record",
  "name": "TradeRaw",
  "namespace": "com.kubetrade.schema.trade",
  "fields": [
    {
      "name": "tradeKey",
      "type": {
        "type": "record",
        "name": "TradeKey",
        "fields": [
          {
            "name": "exchange",
            "type": {
              "type": "enum",
              "name": "Exchange",
              "namespace": "com.kubetrade.schema.common",
              "symbols": [
                "GEMINI"
              ]
            }
          },
          {
            "name": "key",
            "type": "string"
          }
        ]
      }
    },
    {
      "name": "value",
      "type": {
        "type": "record",
        "name": "TradeValue",
        "fields": [
          {
            "name": "exchange",
            "type": "com.kubetrade.schema.common.Exchange"
          },
          {
            "name": "value",
            "type": "string"
          }
        ]
      }
    },
    {
      "name": "symbol",
      "type": "string"
    },
    {
      "name": "payload",
      "type": "string"
    }
  ]
}

Note that for this to happen, all the references have to be appropriately registered in the server. In this context, appropriately registered means that, if I hit the following endpoint for the TradeRaw artifact, I will see the references to the other two artifacts: http://localhost:8080/apis/registry/v2/groups/avro-maven-with-references-auto/artifacts/TradeRaw/versions/2.0/references with result:

[
  {
    "groupId": "avro-maven-with-references-auto",
    "artifactId": "com.kubetrade.schema.trade.TradeKey",
    "version": "1",
    "name": "com.kubetrade.schema.trade.TradeKey"
  },
  {
    "groupId": "avro-maven-with-references-auto",
    "artifactId": "com.kubetrade.schema.trade.TradeValue",
    "version": "1",
    "name": "com.kubetrade.schema.trade.TradeValue"
  }
]

If you give me a more concrete example, I will happily test it, but to me it seems it's working as expected.

[davidradl]

@carlesarnal thanks for your quick response. Can I check what level we need to be on? I assume 2.5.9. I tried to run the sample (I assume by running mvn install) and get: [INFO] --- apicurio-registry-maven-plugin:2.4.3.Final:register (register-artifact) @ apicurio-registry-examples-avro-maven-with-references-auto --- [WARNING] Error processing Avro schema with name TradeKey.avsc. This usually means that the references are not ready yet to parse it [WARNING] Error processing Avro schema with name Exchange.avsc. This usually means that the references are not ready yet to parse it [WARNING] Error processing Avro schema with name TradeValue.avsc. This usually means that the references are not ready yet to parse it [ERROR] Exception while registering artifact [avro-maven-with-references-auto] / [TradeRaw] io.apicurio.registry.rest.client.exception.RestClientException at io.apicurio.registry.rest.client.impl.ErrorHandler.parseError (ErrorHandler.java:95) at io.apicurio.rest.client.JdkHttpClient.sendRequest (JdkHttpClient.java:207) at io.apicurio.registry.rest.client.impl.RegistryClientImpl.createArtifact (RegistryClientImpl.java:328) at io.apicurio.registry.maven.AbstractDirectoryParser.registerArtifact (AbstractDirectoryParser.java:101) at io.apicurio.registry.maven.AbstractDirectoryParser.registerNestedSchema (AbstractDirectoryParser.java:76) at io.apicurio.registry.maven.AvroDirectoryParser.handleSchemaReferences

[davidradl]

@carlesarnal I notice the apicurio-registry-examples repository has been archived. Does this mean it will not be worked on going forward?

[carlesarnal]

@carlesarnal thanks for your quick response. Can I check what level we need to be on? I assume 2.5.9. I tried to run the sample (I assume by running mvn install) and get: [INFO] --- apicurio-registry-maven-plugin:2.4.3.Final:register (register-artifact) @ apicurio-registry-examples-avro-maven-with-references-auto --- [WARNING] Error processing Avro schema with name TradeKey.avsc. This usually means that the references are not ready yet to parse it [WARNING] Error processing Avro schema with name Exchange.avsc. This usually means that the references are not ready yet to parse it [WARNING] Error processing Avro schema with name TradeValue.avsc. This usually means that the references are not ready yet to parse it [ERROR] Exception while registering artifact [avro-maven-with-references-auto] / [TradeRaw] io.apicurio.registry.rest.client.exception.RestClientException at io.apicurio.registry.rest.client.impl.ErrorHandler.parseError (ErrorHandler.java:95) at io.apicurio.rest.client.JdkHttpClient.sendRequest (JdkHttpClient.java:207) at io.apicurio.registry.rest.client.impl.RegistryClientImpl.createArtifact (RegistryClientImpl.java:328) at io.apicurio.registry.maven.AbstractDirectoryParser.registerArtifact (AbstractDirectoryParser.java:101) at io.apicurio.registry.maven.AbstractDirectoryParser.registerNestedSchema (AbstractDirectoryParser.java:76) at io.apicurio.registry.maven.AvroDirectoryParser.handleSchemaReferences

Yes, the error makes sense since the examples repository is not updated. You can update the Apicurio Registry version there to 2.5.9.Final here.

[carlesarnal]

@carlesarnal I notice the apicurio-registry-examples repository has been archived. Does this mean it will not be worked on going forward?

The repository itself no, the examples will for sure continue, you got us in the middle of the Apicurio Registry 3 preparation, and one of the changes is the move of the examples from that repository to the main one, here (there is a note in the readme). I have not pointed you there because Apicurio Registry 3 has not been released yet, so the main branch is a bit unstable right now.

[davidradl]

@carlesarnal I tried to follow https://www.apicur.io/registry/docs/apicurio-registry/2.5.x/getting-started/assembly-managing-registry-artifacts-api.html. I then issue the rest calls:

curl -H "Authorization: Bearer $ACCESS_TOKEN" http://localhost:8085/apis/registry/v2/ids/globalIds/14 { "namespace":"com.example.common", "name":"Item", "type":"record", "fields":[ { "name":"itemId", "type":"com.example.common.ItemId" } ] }% (base) davidradley@davids-mbp-2 bashscripts % curl -H "Authorization: Bearer $ACCESS_TOKEN" http://localhost:8085/apis/registry/v2/ids/globalIds/14?dereference=true zsh: no matches found: http://localhost:8085/apis/registry/v2/ids/globalIds/14?dereference=true

As @seb-pereira indicated the endpoint for dereference=true is not there for me.

I am running:docker run -it -p 8085:8080 apicurio/apicurio-registry-mem:2.5.9.Final on a Mac

[davidradl]

@carlesarnal I tried Yes, the error makes sense since the examples repository is not updated. You can update the Apicurio Registry version there to 2.5.9.Final [here](https://github.com/Apicurio/apicurio-registry-examples/blob/main/pom.xml#L8).

and get The parameters 'existingReferences' for goal io.apicurio:apicurio-registry-maven-plugin:2.5.9.Final:register are missing or invalid

[carlesarnal]

@carlesarnal I tried to follow https://www.apicur.io/registry/docs/apicurio-registry/2.5.x/getting-started/assembly-managing-registry-artifacts-api.html. I then issue the rest calls:

curl -H "Authorization: Bearer $ACCESS_TOKEN" http://localhost:8085/apis/registry/v2/ids/globalIds/14 { "namespace":"com.example.common", "name":"Item", "type":"record", "fields":[ { "name":"itemId", "type":"com.example.common.ItemId" } ] }% (base) davidradley@davids-mbp-2 bashscripts % curl -H "Authorization: Bearer $ACCESS_TOKEN" http://localhost:8085/apis/registry/v2/ids/globalIds/14?dereference=true zsh: no matches found: http://localhost:8085/apis/registry/v2/ids/globalIds/14?dereference=true

As @seb-pereira indicated the endpoint for dereference=true is not there for me.

I am running:docker run -it -p 8085:8080 apicurio/apicurio-registry-mem:2.5.9.Final on a Mac

Ok, I have figured out what's going on. The query parameter has to be escaped for it to work, so if you issue this command you should be all set -> curl http://localhost:8080/apis/registry/v2/ids/globalIds/2?dereference=true.

[carlesarnal]

@carlesarnal I tried Yes, the error makes sense since the examples repository is not updated. You can update the Apicurio Registry version there to 2.5.9.Final [here](https://github.com/Apicurio/apicurio-registry-examples/blob/main/pom.xml#L8).

and get The parameters 'existingReferences' for goal io.apicurio:apicurio-registry-maven-plugin:2.5.9.Final:register are missing or invalid

Sorry, yes, we introduced that parameter and, as I said the example is outdated (and updated on main, but pointing to 3.0). You can fix that problem by adding a line like this to the pom -> https://github.com/Apicurio/apicurio-registry/blob/main/examples/avro-maven-with-references-auto/pom.xml#L44

[davidradl]

@carlesarnal I tried to follow https://www.apicur.io/registry/docs/apicurio-registry/2.5.x/getting-started/assembly-managing-registry-artifacts-api.html. I then issue the rest calls: curl -H "Authorization: Bearer $ACCESS_TOKEN" http://localhost:8085/apis/registry/v2/ids/globalIds/14 { "namespace":"com.example.common", "name":"Item", "type":"record", "fields":[ { "name":"itemId", "type":"com.example.common.ItemId" } ] }% (base) davidradley@davids-mbp-2 bashscripts % curl -H "Authorization: Bearer $ACCESS_TOKEN" http://localhost:8085/apis/registry/v2/ids/globalIds/14?dereference=true zsh: no matches found: http://localhost:8085/apis/registry/v2/ids/globalIds/14?dereference=true As @seb-pereira indicated the endpoint for dereference=true is not there for me. I am running:docker run -it -p 8085:8080 apicurio/apicurio-registry-mem:2.5.9.Final on a Mac

Ok, I have figured out what's going on. The query parameter has to be escaped for it to work, so if you issue this command you should be all set -> curl http://localhost:8080/apis/registry/v2/ids/globalIds/2?dereference=true.

@carlesarnal thankyou with escaping it works for me :-) This is a great circumvention - we need to change the API so that we do not need to escape to add query parameters. Do you know why this is happening? Do you need a new issue for this?

@seb-pereira it looks like we have a way to get the dereferenced schema.

[carlesarnal]

@carlesarnal I tried to follow https://www.apicur.io/registry/docs/apicurio-registry/2.5.x/getting-started/assembly-managing-registry-artifacts-api.html. I then issue the rest calls: curl -H "Authorization: Bearer $ACCESS_TOKEN" http://localhost:8085/apis/registry/v2/ids/globalIds/14 { "namespace":"com.example.common", "name":"Item", "type":"record", "fields":[ { "name":"itemId", "type":"com.example.common.ItemId" } ] }% (base) davidradley@davids-mbp-2 bashscripts % curl -H "Authorization: Bearer $ACCESS_TOKEN" http://localhost:8085/apis/registry/v2/ids/globalIds/14?dereference=true zsh: no matches found: http://localhost:8085/apis/registry/v2/ids/globalIds/14?dereference=true As @seb-pereira indicated the endpoint for dereference=true is not there for me. I am running:docker run -it -p 8085:8080 apicurio/apicurio-registry-mem:2.5.9.Final on a Mac

Ok, I have figured out what's going on. The query parameter has to be escaped for it to work, so if you issue this command you should be all set -> curl http://localhost:8080/apis/registry/v2/ids/globalIds/2?dereference=true.

@carlesarnal thankyou with escaping it works for me :-) This is a great circumvention - we need to change the API so that we do not need to escape to add query parameters. Do you know why this is happening? Do you need a new issue for this?

@seb-pereira it looks like we have a way to get the dereferenced schema.

Sorry, maybe my previous message was not 100% clear. I don't think the API has to change, it's a straight query parameter, it just don't work using curl if the url is not properly encoded, you can either escape it as in my previous message or you can also quote the path and it will just work curl 'http://localhost:8080/apis/registry/v2/ids/globalIds/2?dereference=true'. We're not escaping the parameter in any of our tools. We might need to change the documentation to add the quotes to the URL so it just works in the case of someone following the guide and copy-pasting the command, but not the API for sure.

[davidradl]

@carlesarnal even better! I agree that updating the docs would have helped me.

[seb-pereira]

@carlesarnal @davidradl thanks for your responses. the dereferenced query parameter is ignored with version 2.5.8-final I was using. I confirm it is working as expected with 2.5.9-final. I also managed to register a schema and its references using the ccompat API, and then get the schema with resolved references using Apicurio APIs and the dereferenced query parameter 👍

[carlesarnal]

With Apicurio Registry 2.5.11.Final we have introduced dereferencing support for JSON Schema, OpenAPI and AsynAPI. Now we have support for all the types we think it makes sense. Closing this as the result.