Support schema references for Avro, Protocol Buffer, and JSON schema

[bifrost]

In this pull request, the library has been extended, so the schemas can refer to other registered schemas. The implementation is recursive and it is able to handle multi-level schema references.

The referred schemas are fetched from the schema-registry and are kept in order, so the referred schemas can be processed before the referring schemas. The referred schemas are then processed by the respective schema and schema helpers.

The Avro implementation is using the typeHook and is inspired from https://github.com/kafkajs/confluent-schema-registry/pull/92

Also three new documentation pages for references have been added: schema-json.md, schema-protobuf.md and schema-avro.md

Regards Dan Rasmussen SumUp

[hilleer]

Hi @Nevon

Are you able to assist here or know who could be able to help us in considering this pull request? It seem like an already requested functionality in other pull requests

[Nevon]

I have a review in progress, but not finished yet. Realistically, the soonest I will have enough time available is April 4-8.

I haven't yet looked at how the JSON Schema and Protobuf implementations work, but one thing that stood out to me in reading through the Avro implementation is the usage of typehooks. It's not clear to me how this feature will work when users are already using typehooks to implement logical types, especially in nested schemas. From a cursory look, it doesn't look like the implementation is taking this into account since it doesn't wrap the user-provided typehook.

[bifrost]

Hi @Nevon Could you please provide me with an example of where typehooks are used to implement logical types? Then I will add it to the tests and fix it?

[Nevon]

Sorry, my memory was a bit fuzzy on the details, but you can see an example here where typehooks are used to avoid registering the same type multiple times. The specifics of what the hook is doing is not really important though, but rather that we allow the user to pass in ForSchemaOptions, so we should respect that. The typehook in the test could just as well be just jest.fn().mockReturnValue(<some type>) as long as we see that it gets called and that the return value gets used even if we are using typehooks to implement schema references.

[bifrost]

Hi @Nevon I change the code, so schema references always are parsed for Avro even if a usertypeHook is defined. The typeHook will be called for each avro.Type.forSchema call in the getAvroSchema method. I have also added a test that uses a user-defined typeHook for parsing enums https://github.com/debitoor/confluent-schema-registry/blob/master/src/SchemaRegistry.avro.spec.ts#L440

[bifrost]

Hi @Nevon

when do you have time to have a look at the pull request? The update should handle user-provided typeHook as you requested.

[Nevon]

I'm currently away on vacation until June 25th, so sometime during that week.

[bifrost]

Hi @Nevon Thanks for your review. I have fixed your review comments and added a test for LegacyOptions, please have a look.

[bifrost]

The ts-ignore is needed because the ajv.d.ts does not define addSchema. Ajv extends Core and Core implements addSchema, that is why it works. Plese let me know if I can do a better or how to get types from Core?

[Nevon]

It's not because of Ajv, but because ajv is either an instance of AjvCore or something that implements a similar interface, and that "similar interface" is lacking addSchema. See #163 for context on why this was done.

Changing the JsonOptions type to the following should solve it:

export type JsonOptions = ConstructorParameters<typeof Ajv>[0] & {
  ajvInstance?: {
    addSchema: Ajv['addSchema']
    compile: (schema: any) => ValidateFunction
  }
  referencedSchemas?: JsonConfluentSchema[]
}

[Nevon]

Thank you for the contribution and sorry it took so long to upstream. This is now released in v3.3.0.

[bifrost]

Thanks a lot, Nevon - it has been a pleasure.