"Duplicate type name" error when fetching similar schemas

[bdrobinson]

There is a bug (at least, I think it's unexpected behaviour) in the avsc library where it mutates the passed forSchemaOptions to add a registry to it, which then means subsequent calls using the same forSchemaOptions then crash with a "Duplicate type name" error. See https://github.com/mtth/avsc/issues/312.

This caused a crash for us because the schema registry reuses the options object between calls. So for example, given schemas 1 and 2 which both contain a record with the same name, the following code will crash:

const reg = new SchemaRegistry(
    { host: 'http://localhost:3322' },
    { forSchemaOptions: {} },
);
await reg.getSchema(1);
await reg.getSchema(2); // Error: duplicate type name

Which is obviously quite a big problem because as soon as we try and parse an old and newer version of a subject, our app crashes with this error.

It's worth noting that the crash only happens if we specify forSchemaOptions, as otherwise forSchema gets called with undefined and there's nothing for avsc to mutate.

So primarily I think this is a bug in avsc and I wanted you to be aware – in my opinion it should be perfectly valid for the schema registry to pass the same options into it every time. However it might be worth putting a work around in. For our use case we're patching this line to always pass an empty registry object in:

this.schemasByRegistryId[registryId] = avro.Type.forSchema(schema, {...this.forSchemaOptions, registry: {}})

so that might be a change worth including for the time being? I'm not sure why avsc even exposes the registry as an option as it feels like an implementation detail to me, so perhaps there's some use case I'm not aware of.

Thanks very much!

[ggobbe]

I have the same issue.

While waiting for a fix, my current workaround is to create a separate schemaRegistry instance for every message consumed (far from ideal ofc).

This can also lead to the following scenario regarding producers:

1. register schema A'1 for subject X
2. publish messages with schema A'1 id
3. try to register schema A'2 for subject X, but it is not backward compatible and the subject X compatibility mode is set to backward_transitive
4. get an error saying that the schema is not backward compatible
5. try to register schema A'2 again with the same schema registry instance
6. get a duplicate type name error as the type is already registered in the options.registry in avsc

[danieldunderfelt]

I am also running into this issue when producing messages. It started when timestamp-millis type fields were introduced in our schemas and I implemented the DateType (extending avro.types.LogicalType). Adding this to the config and trying to produce a message containing a timestamp-millis type field crashes the app with the message "ConfluentSchemaRegistryArgumentError: duplicate type name".

I tried to work around this by instantiating separate schema registry clients per topic and not setting the DateType for topics that don't have a timestamp-millis field. This worked, but since then we have introduced timestamp-millis fields on all topics so it is not a valid solution anymore.

Is there any way to fix this?

[henrymunro]

I've also hit this when attempting to pass logicalTypes to the avro options.

Had a go at pushing a PR, making this change locally fixed the issue: https://github.com/kafkajs/confluent-schema-registry/pull/209

[TomCaserta]

@henrymunro I believe you may need to clone the 'registry' property as well if it exists.

For example if you do something like the following:

new SchemaRegistry(
        { host: '...' },
        {
                forSchemaOptions: {
                        logicalTypes: {
                                decimal: DecimalType,
                        },
                        registry: {
                                long: LongType
                        },
                },
        },
);

As otherwise you run into the same issue again since avsc mutates the options.registry property and your cloning doesn't deep clone the options.

On a side note, may look into creating an issue on avsc side as it really should not be mutating this object in my opinion.

[jcjp]

We are actually experiencing this issue but we don't use forSchemaOptions property. We are setting up our schemaRegistry:

public registry = new SchemaRegistry({
    auth: { ... },
    host: { ... }
  }, {
    [SchemaType.AVRO]: {registry: typeMap}
  })

We populate the typeMap variable using this:

import avro from 'avsc'

...

for (let subj of refs) {
      if (!typeMap[subj]) {
        ...
        typeMap[subj] = avro.Type.forSchema(schema)
      }
}

[esahin90]

did somebody try out, if the new version 3.3.0 fixes the issue? It seems like the options passed to the avsc library are now a copy instead of the reference, which prevents the mutation of the options and therefore should fix the duplicate type name error

Update: the issue still remains with 3.3.0

[bdoughertycloudbeds]

This can be used as a workaround: https://github.com/mtth/avsc/issues/294#issuecomment-610176281

In addition to your options (options.registry or options.forSchemaOptions), add typeHook

{
    registry: { 'long': longType },
    typeHook: (schema, opts) => {
      let name = schema.name;
      if (!name) {
        return; // Not a named type, use default logic.
      }

      if (!~name.indexOf('.')) {
        // We need to qualify the type's name.
        const namespace = schema.namespace || opts.namespace;
        if (namespace) {
          name = `${namespace}.${name}`;
        }
      }
      // Return the type registered with the same name, if any.
      return opts.registry[name];
    }
}